Oct 31 02:54:34.308534 ip-10-0-130-12 systemd[1]: Starting Kubernetes Kubelet... Oct 31 02:54:34.737981 ip-10-0-130-12 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:34.737981 ip-10-0-130-12 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 02:54:34.737981 ip-10-0-130-12 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:34.737981 ip-10-0-130-12 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:34.742331 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742228 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.742331 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742327 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.742331 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742333 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.742331 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742337 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742341 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742345 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742349 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742353 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742357 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742360 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742365 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742369 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742373 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742379 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742386 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742392 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742396 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742401 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742406 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742410 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742414 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742418 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.742582 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742422 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742426 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742430 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742434 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742438 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742442 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742446 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742450 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742454 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742458 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742461 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742466 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742470 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742474 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742479 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742484 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742488 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742493 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742497 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742502 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.743360 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742507 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742511 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742515 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742519 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742523 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742528 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742532 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742536 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742540 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742544 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742548 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742553 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742557 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742561 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742565 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742570 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742574 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742578 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742582 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742586 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.744170 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742590 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742594 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742598 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742602 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742606 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742616 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742621 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742625 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742631 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742636 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742640 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742646 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742650 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742654 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742658 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742663 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742667 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742671 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.745014 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742676 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742683 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742690 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742696 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742701 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742705 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742710 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742714 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742718 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742723 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742727 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742731 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742736 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742740 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742744 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742748 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742752 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742756 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742761 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742766 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.745616 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742773 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742777 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742782 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742786 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.742791 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744056 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744066 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744071 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744081 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744085 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744089 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744094 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744098 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744102 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744106 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744110 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744115 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744119 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.746086 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744123 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744127 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744131 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744135 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744140 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744144 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744148 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744152 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744156 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744160 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744164 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744168 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744173 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744178 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744182 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744186 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744190 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744194 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744200 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744205 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.746591 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744209 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744214 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744218 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744222 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744226 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744231 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744235 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744239 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744243 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744269 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744296 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744303 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744307 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744312 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744316 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744321 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744326 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744330 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744334 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.747066 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744338 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744342 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744347 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744351 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744355 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744360 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744364 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744368 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744373 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744377 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744381 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744385 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744390 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744394 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744399 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744403 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744407 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744411 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744415 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744420 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.747563 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744424 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744428 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744432 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744437 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744447 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744451 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744456 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744460 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744464 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744468 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744473 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744479 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744485 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744490 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744494 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744499 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744503 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744507 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744512 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.748020 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744516 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744520 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744525 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744529 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744533 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744537 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744544 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744550 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744555 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744559 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744564 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744568 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744572 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.744576 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745379 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745391 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745394 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745398 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745401 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.748486 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745404 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745407 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745415 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745418 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745421 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745424 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745426 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745430 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745433 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745435 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745438 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745440 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745443 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745445 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745448 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745450 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745453 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745455 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745458 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745460 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.748930 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745463 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745465 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745468 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745470 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745473 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745475 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745478 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745480 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745484 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745487 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745490 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745492 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745495 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745498 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745500 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745503 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745505 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745507 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745510 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745512 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.749402 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745515 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745517 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745522 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745526 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745529 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745532 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745534 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745537 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745539 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745542 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745545 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745547 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745550 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745552 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745555 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745557 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745560 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745562 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745565 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.749873 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745568 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745573 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745576 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745579 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745581 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745584 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745586 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745589 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745591 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745594 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745597 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745599 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745601 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745604 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745606 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745609 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745611 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745616 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745619 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.750326 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745622 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745625 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745627 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745630 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745633 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745636 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745638 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745640 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745643 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745646 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745648 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745651 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745655 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745658 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745661 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745664 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745667 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745669 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745672 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.750795 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745674 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745677 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745679 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.745682 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745779 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745786 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745792 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745796 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745801 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745804 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745808 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745812 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745815 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745818 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745822 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745825 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745828 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745831 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745834 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745837 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745840 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745843 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745848 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 02:54:34.751232 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745850 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745854 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745859 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745862 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745866 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745869 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745872 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745876 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745879 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745882 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745885 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745888 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745891 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745896 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745899 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745902 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745904 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745908 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745911 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745915 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745918 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745921 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745924 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745927 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 02:54:34.751782 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745931 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745934 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745938 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745940 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745943 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745946 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745949 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745952 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745955 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745958 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745962 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745966 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745969 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745972 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745975 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745979 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745983 2576 flags.go:64] FLAG: --help="false" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745986 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745989 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745992 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745994 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.745998 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746001 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 02:54:34.752389 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746004 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746007 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746009 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746012 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746015 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746018 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746021 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746024 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746026 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746029 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746032 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746035 2576 flags.go:64] FLAG: --lock-file="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746038 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746041 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746044 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746049 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746052 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746056 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746059 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746063 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746067 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746070 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746073 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746077 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746080 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 02:54:34.752927 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746084 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746088 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746092 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746095 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746098 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746101 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746104 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746107 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746114 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746117 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746120 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746122 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746125 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746128 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746131 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746134 2576 flags.go:64] FLAG: --port="10250" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746137 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746140 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0f9a2308d368ae1e0" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746143 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746146 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746149 2576 flags.go:64] FLAG: --register-node="true" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746152 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746156 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746159 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746161 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746164 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 02:54:34.753540 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746169 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746173 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746176 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746179 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746181 2576 flags.go:64] FLAG: --runonce="false" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746185 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746187 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746190 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746194 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746197 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746200 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746203 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746206 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746209 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746212 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746214 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746217 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746220 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746223 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746227 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746232 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746235 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746238 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746242 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746258 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 02:54:34.754556 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746261 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746264 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746267 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746287 2576 flags.go:64] FLAG: --v="2" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746293 2576 flags.go:64] FLAG: --version="false" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746297 2576 flags.go:64] FLAG: --vmodule="" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746302 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746307 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746435 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746439 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746443 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746446 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746448 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746451 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746454 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746456 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746459 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746462 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746465 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746469 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746473 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.755543 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746476 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746479 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746482 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746485 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746487 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746490 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746492 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746495 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746497 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746500 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746502 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746505 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746507 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746510 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746512 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746515 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746517 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746520 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746523 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.756114 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746526 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746529 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746532 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746534 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746537 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746539 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746542 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746544 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746547 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746550 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746552 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746555 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746557 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746560 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746562 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746565 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746567 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746569 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746572 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746574 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.756599 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746577 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746579 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746582 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746584 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746586 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746589 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746591 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746594 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746597 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746599 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746602 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746606 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746610 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746612 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746615 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746618 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746620 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746623 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746625 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746628 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.757511 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746630 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746632 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746635 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746638 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746640 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746643 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746645 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746648 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746650 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746653 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746655 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746658 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746660 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746663 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746665 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746668 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746670 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746673 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746675 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746678 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.758407 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746680 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746683 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746686 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746689 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746693 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746696 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746698 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746701 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746703 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746707 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746711 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746713 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746716 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746718 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.746744 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746780 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.759127 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746785 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746788 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746791 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746794 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746796 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746799 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746801 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746803 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746806 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746808 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746811 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746814 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746816 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746819 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746821 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746836 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746839 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746842 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746845 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746847 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.759609 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746852 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746855 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746857 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746860 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746862 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746865 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746868 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746870 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746873 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746875 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746877 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746880 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746883 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746885 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746888 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746890 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746893 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746895 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746898 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746900 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.760294 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746903 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746905 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746907 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746910 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746912 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746915 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746918 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746920 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746922 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746925 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746928 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746930 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746937 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746939 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746942 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746945 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746949 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746952 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.760775 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746954 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746957 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746961 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746964 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746967 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746970 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746973 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746976 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746979 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746981 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746984 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746986 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746989 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746992 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746994 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746996 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.746999 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747001 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747005 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.761236 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747009 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747011 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747014 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747017 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747020 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747022 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747028 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747030 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747033 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747035 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747039 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747042 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747044 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747047 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747049 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747052 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747054 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747057 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747059 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.761764 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747062 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747065 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747068 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747070 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747073 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747076 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747079 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.747081 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.755643 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.755661 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.756580 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.756593 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756702 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756712 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756717 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756722 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756726 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756730 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756735 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.762228 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756739 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756743 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756747 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756752 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756757 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756761 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756765 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756769 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756773 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756777 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756788 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756793 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756797 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756802 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756806 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756810 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756814 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756818 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756822 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.762727 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756826 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756831 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756835 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756840 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756845 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756849 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756854 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756858 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756862 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756867 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756871 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756876 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756880 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756884 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756888 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756892 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756896 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756901 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756904 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756909 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.763157 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756914 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756918 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756923 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756927 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756931 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756935 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756939 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756943 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756947 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756952 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756956 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756961 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756965 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756969 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756974 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756978 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756982 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756987 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756991 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.756996 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.763788 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757000 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757004 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757009 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757013 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757017 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757021 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757026 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757030 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757034 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757038 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757042 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757047 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757051 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757057 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757061 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757070 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757074 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757078 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757082 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.764282 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757087 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757091 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757095 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757100 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757104 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757109 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757116 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757122 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757128 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757134 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757140 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757145 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757151 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757155 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757160 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757165 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757169 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757174 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.764887 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757178 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757183 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.757223 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757523 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757534 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757539 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757544 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757549 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757555 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757559 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757563 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757568 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757572 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757577 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757581 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757585 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.765530 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757589 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757593 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757598 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757603 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757607 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757612 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757616 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757620 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757624 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757629 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757633 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757637 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757642 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757646 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757651 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757655 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757659 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757663 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757667 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757672 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.765912 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757676 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757681 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757685 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757689 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757693 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757698 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757703 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757707 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757711 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757716 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757720 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757727 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757734 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757741 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757746 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757750 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757755 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757759 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757763 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.766428 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757768 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757772 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757776 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757780 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757784 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757788 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757792 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757796 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757801 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757805 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757809 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757813 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757818 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757822 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757826 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757830 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757834 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757838 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.766866 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757847 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757851 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757856 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757860 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757864 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757868 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757873 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757876 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757880 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757885 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757889 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757893 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757897 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757902 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757906 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757910 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757914 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757918 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757922 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757926 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.767347 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757930 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757934 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757938 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757943 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757951 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757955 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757959 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757963 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757967 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757971 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757975 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757979 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757987 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.757991 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.758030 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:34.767893 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758080 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758086 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758091 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758096 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758100 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758104 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758109 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758113 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758118 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758122 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758126 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758130 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758134 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758138 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758142 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758146 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758150 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758155 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758159 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758163 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:34.768418 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758167 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758171 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758175 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758180 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758184 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758188 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758192 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758196 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758200 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758207 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758212 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758217 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758221 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758225 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758230 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758234 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758238 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758242 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758263 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758268 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:34.768896 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758272 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758276 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758279 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758284 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758288 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758292 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758296 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758301 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758305 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758309 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758313 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758318 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758322 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758326 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758330 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758334 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758339 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758343 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758347 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758351 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:34.769539 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758355 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758361 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758365 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758371 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758375 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758379 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758384 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758388 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758394 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758398 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758402 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758406 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758410 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758417 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758424 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758429 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758435 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758440 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:34.770359 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758445 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758449 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758453 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758458 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758462 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758466 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758470 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758474 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758479 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758483 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758487 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758492 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758496 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758500 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758504 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758509 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758514 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758519 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758524 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758528 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:34.770782 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758532 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758536 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758541 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758545 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758549 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758553 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758557 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:34.758564 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.758638 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.766145 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.767275 2576 server.go:1014] "Starting client certificate rotation" Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.767375 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.767410 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.769762 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:54:34.771338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.770828 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:54:34.782360 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.782342 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 02:54:34.786452 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.786138 2576 log.go:25] "Validated CRI v1 image API" Oct 31 02:54:34.787040 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.787015 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 02:54:34.791929 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.791910 2576 fs.go:135] Filesystem UUIDs: map[15d6fb13-90ef-421b-be10-1fea330abcf6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fcfed4ff-01a3-45ff-8067-e1f6f3845409:/dev/nvme0n1p3] Oct 31 02:54:34.792004 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.791928 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 02:54:34.797434 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.797317 2576 manager.go:217] Machine: {Timestamp:2026-10-31 02:54:34.795854369 +0000 UTC m=+0.374270324 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100513 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d67836b784544d88235e5caddba0b SystemUUID:ec2d6783-6b78-4544-d882-35e5caddba0b BootID:2412d30f-e801-411b-8699-be74b247ab13 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:0f:4f:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:0f:4f:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:ac:28:f1:6d:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 02:54:34.797434 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.797433 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 02:54:34.797554 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.797542 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 02:54:34.801469 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.801440 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 02:54:34.801607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.801472 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-12.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 02:54:34.801676 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.801605 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 02:54:34.802272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.802243 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 02:54:34.802311 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.802274 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 02:54:34.802311 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.802304 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:54:34.802364 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.802317 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:54:34.803598 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.803588 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 02:54:34.808227 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808215 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 02:54:34.808355 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808346 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 02:54:34.808395 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808379 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 02:54:34.808395 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808388 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 02:54:34.808458 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808398 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 02:54:34.808458 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808406 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 02:54:34.808458 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808414 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 02:54:34.808742 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.808729 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:54:34.811299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.811281 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 02:54:34.818533 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.818517 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:54:34.819461 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819438 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 02:54:34.819513 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819469 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 02:54:34.819626 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819616 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 02:54:34.819659 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819632 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 02:54:34.819659 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819638 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 02:54:34.819659 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819643 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 02:54:34.819659 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819650 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 02:54:34.819659 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.819657 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 02:54:34.820376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.820367 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 02:54:34.820449 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.820377 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 02:54:34.820449 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.820384 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 02:54:34.820449 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.820390 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 02:54:34.820449 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.820398 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 02:54:34.820449 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.820407 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 02:54:34.821170 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.821160 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 02:54:34.821213 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.821173 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 02:54:34.824276 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.824210 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-12.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 02:54:34.824958 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.824943 2576 server.go:1263] "Started kubelet" Oct 31 02:54:34.825039 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.824997 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 02:54:34.825145 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.825104 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 02:54:34.825189 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.825165 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 02:54:34.825603 ip-10-0-130-12 systemd[1]: Started Kubernetes Kubelet. Oct 31 02:54:34.826619 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.826601 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 02:54:34.827000 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.826982 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 02:54:34.830441 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.830417 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 02:54:34.834242 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.834224 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 02:54:34.834242 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.834243 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 02:54:34.834384 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.834310 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:34.834957 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.834926 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 02:54:34.835066 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835049 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 02:54:34.835066 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.834942 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 02:54:34.835214 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835122 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 02:54:34.835214 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835130 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 02:54:34.835364 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835353 2576 factory.go:55] Registering systemd factory Oct 31 02:54:34.835406 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835367 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 02:54:34.835569 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835553 2576 factory.go:153] Registering CRI-O factory Oct 31 02:54:34.835628 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.835575 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 02:54:34.836109 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.836092 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 02:54:34.836166 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.836121 2576 factory.go:103] Registering Raw factory Oct 31 02:54:34.836166 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.836136 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 02:54:34.837185 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.837172 2576 manager.go:319] Starting recovery of all containers Oct 31 02:54:34.844042 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.844014 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-12.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 02:54:34.844195 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.844174 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 02:54:34.844500 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.844467 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-12.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 02:54:34.844743 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.841545 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-12.us-east-2.compute.internal.18e37db0c6b2dedc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-12.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-12.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:34.8249167 +0000 UTC m=+0.403332654,LastTimestamp:2026-10-31 02:54:34.8249167 +0000 UTC m=+0.403332654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-12.us-east-2.compute.internal,}" Oct 31 02:54:34.849718 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.848848 2576 manager.go:324] Recovery completed Oct 31 02:54:34.849718 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.849272 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 02:54:34.850697 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.850660 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 02:54:34.853461 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.853448 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:34.855102 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855082 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:34.855149 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855119 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:34.855149 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855130 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:34.855638 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855626 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:34.855695 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855627 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 02:54:34.855695 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855656 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 02:54:34.855695 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.855674 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 02:54:34.856288 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.856265 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:34.856358 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.856296 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:34.856358 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.856312 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:34.857895 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.857883 2576 policy_none.go:50] "Start" Oct 31 02:54:34.857949 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.857900 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 02:54:34.857949 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.857910 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 02:54:34.858384 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.858323 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-12.us-east-2.compute.internal.18e37db0c87f79ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-12.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-12.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-12.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:34.855102959 +0000 UTC m=+0.433518912,LastTimestamp:2026-10-31 02:54:34.855102959 +0000 UTC m=+0.433518912,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-12.us-east-2.compute.internal,}" Oct 31 02:54:34.859895 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.859884 2576 policy_none.go:44] "Start" Oct 31 02:54:34.861220 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.861165 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-12.us-east-2.compute.internal.18e37db0c87fcd57 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-12.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-12.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-12.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:34.855124311 +0000 UTC m=+0.433540265,LastTimestamp:2026-10-31 02:54:34.855124311 +0000 UTC m=+0.433540265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-12.us-east-2.compute.internal,}" Oct 31 02:54:34.870875 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.870858 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f7kth" Oct 31 02:54:34.872194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.872135 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-12.us-east-2.compute.internal.18e37db0c87ff4e2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-12.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-12.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-12.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:34.855134434 +0000 UTC m=+0.433550388,LastTimestamp:2026-10-31 02:54:34.855134434 +0000 UTC m=+0.433550388,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-12.us-east-2.compute.internal,}" Oct 31 02:54:34.877602 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.877584 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f7kth" Oct 31 02:54:34.882286 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.882071 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-12.us-east-2.compute.internal.18e37db0c87f79ef\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-12.us-east-2.compute.internal.18e37db0c87f79ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-12.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-12.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-12.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:34.855102959 +0000 UTC m=+0.433518912,LastTimestamp:2026-10-31 02:54:34.856282806 +0000 UTC m=+0.434698764,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-12.us-east-2.compute.internal,}" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.910794 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913216 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.913266 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913279 2576 server.go:87] "Starting device plugin registration server" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913498 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913509 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913605 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913694 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.913703 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.914346 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.914391 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.914744 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.916023 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.916041 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.916056 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 02:54:34.918376 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:34.916088 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 02:54:34.920748 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:34.920293 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:35.014208 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.014168 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:35.016500 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.016480 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal"] Oct 31 02:54:35.016547 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.016529 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:35.019995 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.019978 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:35.020053 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.020011 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:35.020053 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.020021 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:35.020120 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.019978 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:35.020120 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.020080 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:35.020120 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.020094 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:35.020120 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.020114 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.022418 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.022407 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:35.022560 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.022545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.022598 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.022577 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:35.023141 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.023123 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:35.023220 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.023123 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:35.023220 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.023179 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:35.023220 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.023192 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:35.023220 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.023153 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:35.023388 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.023222 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:35.025387 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.025373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.025446 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.025397 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:35.026737 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.026344 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:35.026737 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.026369 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:35.026737 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.026383 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:35.031674 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.031660 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.031730 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.031680 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-12.us-east-2.compute.internal\": node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.036411 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.036391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.036506 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.036422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.036506 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.036438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/915a6fb768d2fd4c56c8c1c0d6da80b2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"915a6fb768d2fd4c56c8c1c0d6da80b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.045784 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.045768 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.047726 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.047712 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.063330 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.063311 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.137345 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.137326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.137392 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.137356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.137392 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.137376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/915a6fb768d2fd4c56c8c1c0d6da80b2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"915a6fb768d2fd4c56c8c1c0d6da80b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.137459 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.137406 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/915a6fb768d2fd4c56c8c1c0d6da80b2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"915a6fb768d2fd4c56c8c1c0d6da80b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.137459 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.137422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.137459 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.137433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.163813 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.163795 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.264380 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.264332 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.351187 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.351167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.353334 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.353315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:35.364820 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.364798 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.465363 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.465339 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.565932 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.565881 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.666525 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.666500 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.767068 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.767044 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.767481 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.767078 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 02:54:35.767481 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.767222 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 02:54:35.867603 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.867576 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:35.883349 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.883328 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:35.885400 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.885376 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 02:49:35 +0000 UTC" deadline="2028-06-17 12:06:10.111649475 +0000 UTC" Oct 31 02:54:35.885460 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.885400 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14289h11m34.226251864s" Oct 31 02:54:35.920642 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.919393 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:35.921050 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.921030 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:35.921127 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.921069 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:35.921127 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.921085 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:35.945514 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:35.945487 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915a6fb768d2fd4c56c8c1c0d6da80b2.slice/crio-7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6 WatchSource:0}: Error finding container 7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6: Status 404 returned error can't find the container with id 7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6 Oct 31 02:54:35.949669 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:35.949647 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9031209d866497c6611ade6539778f.slice/crio-f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc WatchSource:0}: Error finding container f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc: Status 404 returned error can't find the container with id f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc Oct 31 02:54:35.951579 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:35.951563 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:54:35.968361 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:35.968343 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:36.027200 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.027177 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:36.068456 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.068433 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:54:36.094328 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.094287 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:36.135354 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.135336 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:36.150212 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.150196 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:54:36.151241 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.151230 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:54:36.160578 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.160565 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:54:36.809800 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.809732 2576 apiserver.go:52] "Watching apiserver" Oct 31 02:54:36.818536 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.818509 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 02:54:36.818931 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.818902 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5nsnk","openshift-multus/multus-additional-cni-plugins-qrb4n","openshift-multus/network-metrics-daemon-5qlrx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal","openshift-multus/multus-9rzxm","openshift-network-diagnostics/network-check-target-r76hd","openshift-network-operator/iptables-alerter-jpxqb","kube-system/konnectivity-agent-sjjkf","kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn","openshift-cluster-node-tuning-operator/tuned-7vws7","openshift-dns/node-resolver-2ltlq","openshift-image-registry/node-ca-8nv7d","kube-system/global-pull-secret-syncer-4pp9z"] Oct 31 02:54:36.819477 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.819448 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:36.821575 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.821555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.824074 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.824056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.826029 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.826009 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 02:54:36.826123 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.826069 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 02:54:36.826196 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.826189 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wcv7f\"" Oct 31 02:54:36.826519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.826499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:36.826628 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.826583 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:36.827973 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.827951 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.827973 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.827962 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 02:54:36.828630 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.828609 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wr2d\"" Oct 31 02:54:36.829143 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.828778 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.829143 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.828872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.831172 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.831153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.833424 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.833403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:36.833524 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.833473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:36.835374 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.835001 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 02:54:36.835957 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.835929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.838833 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.838815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.841313 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.841295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.843835 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.843812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:36.845174 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-netd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.845277 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qbt\" (UniqueName: \"kubernetes.io/projected/b6bfe2e7-b724-42de-acff-a45a1609c7d0-kube-api-access-95qbt\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.845277 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.845454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-var-lib-kubelet\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.845454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-multus\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.845454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-cnibin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.845454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-etc-kubernetes\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.845454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphtw\" (UniqueName: \"kubernetes.io/projected/9888db69-6183-403a-82cf-01ad3667c6f9-kube-api-access-pphtw\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.845709 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b408b194-9df2-4518-b247-0d9a83d80f3b-serviceca\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.845709 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:36.845709 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-systemd-units\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.845930 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-konnectivity-ca\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.845996 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-conf\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.845996 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-sys\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.845996 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.845983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-lib-modules\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.846153 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-kubelet\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.846153 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846153 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-modprobe-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.846153 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-node-log\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846370 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-log-socket\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846370 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.846370 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-bin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.846528 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txxr\" (UniqueName: \"kubernetes.io/projected/b408b194-9df2-4518-b247-0d9a83d80f3b-kube-api-access-5txxr\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.846528 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846528 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b408b194-9df2-4518-b247-0d9a83d80f3b-host\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.846528 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846507 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-var-lib-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846736 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-script-lib\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846736 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-netns\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846736 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-systemd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846736 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-etc-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846736 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-host\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-conf-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxbmw\" (UniqueName: \"kubernetes.io/projected/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-kube-api-access-xxbmw\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-ovn\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-k8s-cni-cncf-io\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-system-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.846985 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.846956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-multus-daemon-config\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.847761 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-agent-certs\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.847761 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xb6\" (UniqueName: \"kubernetes.io/projected/111372d5-9b74-4bae-b96f-3671787ad5b3-kube-api-access-m5xb6\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:36.847761 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-run\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.847761 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-tmp\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.847989 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-multus-certs\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.847989 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-bin\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.847989 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-cni-binary-copy\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.847989 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-iptables-alerter-script\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.847989 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-config\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.847989 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.847964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf9a703-8b57-41ae-a801-cee51331a479-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6p6\" (UniqueName: \"kubernetes.io/projected/dcf9a703-8b57-41ae-a801-cee51331a479-kube-api-access-5n6p6\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-kubernetes\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-systemd\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-tuned\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-socket-dir-parent\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-host-slash\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-env-overrides\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.848294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-kubelet\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.848736 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-slash\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.848794 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysconfig\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.848794 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-netns\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.848894 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.848814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-hostroot\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.851145 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.850867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.852504 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.852482 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 02:54:36.855435 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.855169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:36.855435 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.855238 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:36.883420 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.883399 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-59cqs" Oct 31 02:54:36.892385 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.892334 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-59cqs" Oct 31 02:54:36.923803 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.923755 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"ea9031209d866497c6611ade6539778f","Type":"ContainerStarted","Data":"f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc"} Oct 31 02:54:36.925224 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.925197 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"915a6fb768d2fd4c56c8c1c0d6da80b2","Type":"ContainerStarted","Data":"7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6"} Oct 31 02:54:36.934934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.934913 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.935219 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.935180 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.935219 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.935216 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 02:54:36.935404 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.935329 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.935468 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.935410 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 02:54:36.935468 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.935181 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 02:54:36.935677 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.935660 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.936220 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936204 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 02:54:36.936401 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936381 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls2x2\"" Oct 31 02:54:36.936511 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936402 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 02:54:36.936569 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936513 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 02:54:36.936628 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936588 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sts47\"" Oct 31 02:54:36.936730 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936714 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x5rg4\"" Oct 31 02:54:36.936784 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936717 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 02:54:36.936784 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936744 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.936784 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936771 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.936936 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936890 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 02:54:36.936987 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.936964 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.937541 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937526 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 02:54:36.937633 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937576 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.937786 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937771 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.937864 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937812 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.937928 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937880 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-49z5f\"" Oct 31 02:54:36.937987 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937947 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 02:54:36.938044 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.937986 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 02:54:36.938044 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.938025 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 02:54:36.938147 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.938057 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:54:36.938197 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.938153 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvhtj\"" Oct 31 02:54:36.938451 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.938435 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2b67f\"" Oct 31 02:54:36.938553 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.938503 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbpkr\"" Oct 31 02:54:36.949832 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.949809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txxr\" (UniqueName: \"kubernetes.io/projected/b408b194-9df2-4518-b247-0d9a83d80f3b-kube-api-access-5txxr\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.949941 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.949849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.949941 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.949884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-etc-selinux\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.949941 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.949911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b408b194-9df2-4518-b247-0d9a83d80f3b-host\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.949941 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.949938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-var-lib-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.949964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-script-lib\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-netns\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-systemd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-etc-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-host\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-conf-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxbmw\" (UniqueName: \"kubernetes.io/projected/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-kube-api-access-xxbmw\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.950157 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-ovn\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-k8s-cni-cncf-io\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-system-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-netns\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-systemd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-multus-daemon-config\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950331 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-kubelet-config\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e948ca8-1809-4eeb-b014-f563cb6fdb91-tmp-dir\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-agent-certs\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xb6\" (UniqueName: \"kubernetes.io/projected/111372d5-9b74-4bae-b96f-3671787ad5b3-kube-api-access-m5xb6\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-run\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-tmp\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-multus-certs\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-socket-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.950651 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-bin\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e948ca8-1809-4eeb-b014-f563cb6fdb91-hosts-file\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-cni-binary-copy\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-iptables-alerter-script\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-config\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf9a703-8b57-41ae-a801-cee51331a479-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6p6\" (UniqueName: \"kubernetes.io/projected/dcf9a703-8b57-41ae-a801-cee51331a479-kube-api-access-5n6p6\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-kubernetes\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-systemd\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-tuned\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-socket-dir-parent\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-multus-daemon-config\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-host-slash\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-env-overrides\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.951519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950967 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-etc-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-registration-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-kubelet\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-slash\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.951057 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysconfig\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-netns\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.951142 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:37.451114962 +0000 UTC m=+3.029530909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-netns\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-hostroot\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951235 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-hostroot\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951284 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-host\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951352 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b408b194-9df2-4518-b247-0d9a83d80f3b-host\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951398 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-var-lib-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951459 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-ovn\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.950928 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-host-slash\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-multus-certs\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.952299 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-slash\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.951980 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-bin\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-kubelet\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snh2\" (UniqueName: \"kubernetes.io/projected/5e948ca8-1809-4eeb-b014-f563cb6fdb91-kube-api-access-6snh2\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cq9z\" (UniqueName: \"kubernetes.io/projected/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-kube-api-access-5cq9z\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-k8s-cni-cncf-io\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-netd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qbt\" (UniqueName: \"kubernetes.io/projected/b6bfe2e7-b724-42de-acff-a45a1609c7d0-kube-api-access-95qbt\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952175 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-system-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-systemd\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-socket-dir-parent\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-run\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-var-lib-kubelet\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-multus\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-cnibin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-etc-kubernetes\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953144 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphtw\" (UniqueName: \"kubernetes.io/projected/9888db69-6183-403a-82cf-01ad3667c6f9-kube-api-access-pphtw\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952621 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-env-overrides\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b408b194-9df2-4518-b247-0d9a83d80f3b-serviceca\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-systemd-units\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-konnectivity-ca\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-conf\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-sys\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-lib-modules\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-script-lib\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.952993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-kubelet\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-config\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-modprobe-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953030 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-cni-binary-copy\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-dbus\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:36.953942 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953121 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysconfig\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953231 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-lib-modules\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-kubernetes\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-kubelet\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-conf-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-modprobe-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-etc-kubernetes\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-device-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953546 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-cnibin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rvq\" (UniqueName: \"kubernetes.io/projected/22caca16-dc54-4bac-8dff-b87cab1b8b60-kube-api-access-g4rvq\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cnibin\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-node-log\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-log-socket\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-sys-fs\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:36.955013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953736 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-sys\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-bin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-systemd-units\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953827 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-netd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953856 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-bin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-log-socket\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953893 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-iptables-alerter-script\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953885 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-konnectivity-ca\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.953955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-var-lib-kubelet\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.954040 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-conf\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.954096 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-multus\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.954142 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-node-log\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.954230 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.954441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.955914 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.954470 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b408b194-9df2-4518-b247-0d9a83d80f3b-serviceca\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.961773 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.956819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-tmp\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.961773 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.956843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-tuned\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.961773 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.956859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf9a703-8b57-41ae-a801-cee51331a479-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:36.961773 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.957171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-agent-certs\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:36.964746 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.964459 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxbmw\" (UniqueName: \"kubernetes.io/projected/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-kube-api-access-xxbmw\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:36.969674 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.965372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txxr\" (UniqueName: \"kubernetes.io/projected/b408b194-9df2-4518-b247-0d9a83d80f3b-kube-api-access-5txxr\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:36.969674 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.967124 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:36.969674 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.967147 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:36.969674 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.967162 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6x7mb for pod openshift-network-diagnostics/network-check-target-r76hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:36.969674 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:36.967237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb podName:6aa634f6-8f9c-4cd6-946d-bc018c3c20bc nodeName:}" failed. No retries permitted until 2026-10-31 02:54:37.467214927 +0000 UTC m=+3.045631306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6x7mb" (UniqueName: "kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb") pod "network-check-target-r76hd" (UID: "6aa634f6-8f9c-4cd6-946d-bc018c3c20bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:36.969674 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.969511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xb6\" (UniqueName: \"kubernetes.io/projected/111372d5-9b74-4bae-b96f-3671787ad5b3-kube-api-access-m5xb6\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:36.970581 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.970486 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphtw\" (UniqueName: \"kubernetes.io/projected/9888db69-6183-403a-82cf-01ad3667c6f9-kube-api-access-pphtw\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:54:36.971699 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.971654 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qbt\" (UniqueName: \"kubernetes.io/projected/b6bfe2e7-b724-42de-acff-a45a1609c7d0-kube-api-access-95qbt\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:36.972715 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:36.972687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6p6\" (UniqueName: \"kubernetes.io/projected/dcf9a703-8b57-41ae-a801-cee51331a479-kube-api-access-5n6p6\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:37.054479 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-kubelet-config\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.054479 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e948ca8-1809-4eeb-b014-f563cb6fdb91-tmp-dir\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.054677 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.054677 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-socket-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.054677 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.054677 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e948ca8-1809-4eeb-b014-f563cb6fdb91-hosts-file\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.054677 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.054620 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:37.054677 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054676 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-socket-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-kubelet-config\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e948ca8-1809-4eeb-b014-f563cb6fdb91-hosts-file\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054894 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e948ca8-1809-4eeb-b014-f563cb6fdb91-tmp-dir\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-registration-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.054934 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snh2\" (UniqueName: \"kubernetes.io/projected/5e948ca8-1809-4eeb-b014-f563cb6fdb91-kube-api-access-6snh2\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cq9z\" (UniqueName: \"kubernetes.io/projected/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-kube-api-access-5cq9z\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-registration-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.054987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-dbus\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-device-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rvq\" (UniqueName: \"kubernetes.io/projected/22caca16-dc54-4bac-8dff-b87cab1b8b60-kube-api-access-g4rvq\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cnibin\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-sys-fs\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-etc-selinux\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-device-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.055426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-etc-selinux\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055431 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-sys-fs\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055535 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cnibin\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.055579 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret podName:566be63b-5b2b-4f4f-a40f-1e83e7837f74 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:37.555554861 +0000 UTC m=+3.133970821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret") pod "global-pull-secret-syncer-4pp9z" (UID: "566be63b-5b2b-4f4f-a40f-1e83e7837f74") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-dbus\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055634 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.055935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.056062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.056023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.071329 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.071270 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snh2\" (UniqueName: \"kubernetes.io/projected/5e948ca8-1809-4eeb-b014-f563cb6fdb91-kube-api-access-6snh2\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.072600 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.072569 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rvq\" (UniqueName: \"kubernetes.io/projected/22caca16-dc54-4bac-8dff-b87cab1b8b60-kube-api-access-g4rvq\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.073201 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.073173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cq9z\" (UniqueName: \"kubernetes.io/projected/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-kube-api-access-5cq9z\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.138682 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.138644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:37.147093 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.146894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:54:37.150044 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.149482 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b281bc_7922_46b5_8d33_2f1bfdebb8ec.slice/crio-e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03 WatchSource:0}: Error finding container e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03: Status 404 returned error can't find the container with id e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03 Oct 31 02:54:37.156674 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.156646 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb408b194_9df2_4518_b247_0d9a83d80f3b.slice/crio-c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061 WatchSource:0}: Error finding container c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061: Status 404 returned error can't find the container with id c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061 Oct 31 02:54:37.158289 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.158246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:54:37.167062 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.167038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:54:37.168505 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.168477 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bfe2e7_b724_42de_acff_a45a1609c7d0.slice/crio-67d6e1817baeeb75090434a01af43e152c7964278d20b9eb6d3984dc00ea05fc WatchSource:0}: Error finding container 67d6e1817baeeb75090434a01af43e152c7964278d20b9eb6d3984dc00ea05fc: Status 404 returned error can't find the container with id 67d6e1817baeeb75090434a01af43e152c7964278d20b9eb6d3984dc00ea05fc Oct 31 02:54:37.177948 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.177924 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d72e18_bf2d_46e9_91ef_2374a0d41cf1.slice/crio-7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280 WatchSource:0}: Error finding container 7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280: Status 404 returned error can't find the container with id 7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280 Oct 31 02:54:37.188901 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.188877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9rzxm" Oct 31 02:54:37.194023 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.194003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:37.200331 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.200315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:54:37.202867 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.202714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:54:37.204414 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.204385 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf9a703_8b57_41ae_a801_cee51331a479.slice/crio-c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3 WatchSource:0}: Error finding container c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3: Status 404 returned error can't find the container with id c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3 Oct 31 02:54:37.213596 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.213569 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22caca16_dc54_4bac_8dff_b87cab1b8b60.slice/crio-24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111 WatchSource:0}: Error finding container 24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111: Status 404 returned error can't find the container with id 24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111 Oct 31 02:54:37.217739 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.217719 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e948ca8_1809_4eeb_b014_f563cb6fdb91.slice/crio-c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7 WatchSource:0}: Error finding container c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7: Status 404 returned error can't find the container with id c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7 Oct 31 02:54:37.224388 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.224349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:54:37.236343 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:54:37.236301 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd31a28_7ec3_4d09_ac91_e060a5e2d56e.slice/crio-86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d WatchSource:0}: Error finding container 86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d: Status 404 returned error can't find the container with id 86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d Oct 31 02:54:37.458520 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.458438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:37.458657 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.458604 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:37.458714 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.458672 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:38.458652207 +0000 UTC m=+4.037068152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:37.558844 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.558803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:37.559025 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.558893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.559025 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.559012 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:37.559158 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.559038 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:37.559158 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.559049 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6x7mb for pod openshift-network-diagnostics/network-check-target-r76hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:37.559158 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.559095 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb podName:6aa634f6-8f9c-4cd6-946d-bc018c3c20bc nodeName:}" failed. No retries permitted until 2026-10-31 02:54:38.559081444 +0000 UTC m=+4.137497408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6x7mb" (UniqueName: "kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb") pod "network-check-target-r76hd" (UID: "6aa634f6-8f9c-4cd6-946d-bc018c3c20bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:37.559344 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.559014 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:37.559469 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.559394 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret podName:566be63b-5b2b-4f4f-a40f-1e83e7837f74 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:38.55937543 +0000 UTC m=+4.137791374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret") pod "global-pull-secret-syncer-4pp9z" (UID: "566be63b-5b2b-4f4f-a40f-1e83e7837f74") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:37.897108 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.895499 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:37 +0000 UTC" deadline="2028-06-15 03:39:45.890159938 +0000 UTC" Oct 31 02:54:37.897108 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.895542 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14232h45m7.994623292s" Oct 31 02:54:37.917336 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.916828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:37.917336 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:37.916949 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:37.929246 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.929145 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerStarted","Data":"86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d"} Oct 31 02:54:37.940802 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.937173 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ltlq" event={"ID":"5e948ca8-1809-4eeb-b014-f563cb6fdb91","Type":"ContainerStarted","Data":"c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7"} Oct 31 02:54:37.945598 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.945494 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" event={"ID":"22caca16-dc54-4bac-8dff-b87cab1b8b60","Type":"ContainerStarted","Data":"24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111"} Oct 31 02:54:37.955572 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.955504 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9rzxm" event={"ID":"9888db69-6183-403a-82cf-01ad3667c6f9","Type":"ContainerStarted","Data":"6fbdd3ce7a3372e444b27cf74755411e927d85ddfb8d1e05be748a075d7cd5b2"} Oct 31 02:54:37.958338 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.958257 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jpxqb" event={"ID":"70d72e18-bf2d-46e9-91ef-2374a0d41cf1","Type":"ContainerStarted","Data":"7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280"} Oct 31 02:54:37.960351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.960193 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3"} Oct 31 02:54:37.966488 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.966461 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vws7" event={"ID":"b6bfe2e7-b724-42de-acff-a45a1609c7d0","Type":"ContainerStarted","Data":"67d6e1817baeeb75090434a01af43e152c7964278d20b9eb6d3984dc00ea05fc"} Oct 31 02:54:37.968899 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.968566 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8nv7d" event={"ID":"b408b194-9df2-4518-b247-0d9a83d80f3b","Type":"ContainerStarted","Data":"c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061"} Oct 31 02:54:37.971368 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:37.970063 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sjjkf" event={"ID":"64b281bc-7922-46b5-8d33-2f1bfdebb8ec","Type":"ContainerStarted","Data":"e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03"} Oct 31 02:54:38.472224 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.471995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:38.472356 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.472190 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:38.472356 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.472354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:40.472329994 +0000 UTC m=+6.050745940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:38.572873 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.572842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:38.573013 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.572909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:38.573091 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.573072 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:38.573091 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.573090 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:38.573202 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.573104 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6x7mb for pod openshift-network-diagnostics/network-check-target-r76hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:38.573202 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.573161 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb podName:6aa634f6-8f9c-4cd6-946d-bc018c3c20bc nodeName:}" failed. No retries permitted until 2026-10-31 02:54:40.573142696 +0000 UTC m=+6.151558642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6x7mb" (UniqueName: "kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb") pod "network-check-target-r76hd" (UID: "6aa634f6-8f9c-4cd6-946d-bc018c3c20bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:38.573607 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.573587 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:38.573723 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.573648 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret podName:566be63b-5b2b-4f4f-a40f-1e83e7837f74 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:40.573630433 +0000 UTC m=+6.152046393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret") pod "global-pull-secret-syncer-4pp9z" (UID: "566be63b-5b2b-4f4f-a40f-1e83e7837f74") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:38.896669 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.896598 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:37 +0000 UTC" deadline="2028-06-03 20:57:29.51948791 +0000 UTC" Oct 31 02:54:38.896669 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.896665 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13962h2m50.622827281s" Oct 31 02:54:38.919504 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.919471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:38.919983 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.919612 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:38.920145 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.920108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:38.920250 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:38.920220 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:38.994793 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.994698 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea9031209d866497c6611ade6539778f" containerID="af4d86591d77c4052810d072d3b8224fcca83f37fa6cab3ecd751a84552f8238" exitCode=0 Oct 31 02:54:38.994954 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:38.994792 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"ea9031209d866497c6611ade6539778f","Type":"ContainerDied","Data":"af4d86591d77c4052810d072d3b8224fcca83f37fa6cab3ecd751a84552f8238"} Oct 31 02:54:39.016620 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:39.016052 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"915a6fb768d2fd4c56c8c1c0d6da80b2","Type":"ContainerStarted","Data":"1a35190fb070366149190d749e50fb30895d2ad2e158c08243c0eb7b0bdabc9a"} Oct 31 02:54:39.917213 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:39.917180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:39.917418 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:39.917332 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:40.039278 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.039222 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"ea9031209d866497c6611ade6539778f","Type":"ContainerStarted","Data":"e629a732b0670531ba396298993c4a1e8deed62d54a3e32eaef44c24bbaf4b1a"} Oct 31 02:54:40.059690 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.059635 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" podStartSLOduration=4.059616083 podStartE2EDuration="4.059616083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:39.035479127 +0000 UTC m=+4.613895095" watchObservedRunningTime="2026-10-31 02:54:40.059616083 +0000 UTC m=+5.638032050" Oct 31 02:54:40.490304 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.489661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:40.490304 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.489844 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:40.490304 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.489917 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:44.489896369 +0000 UTC m=+10.068312326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:40.591028 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.590943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:40.591028 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.591014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:40.591202 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.591187 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:40.591263 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.591211 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:40.591263 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.591225 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6x7mb for pod openshift-network-diagnostics/network-check-target-r76hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:40.591383 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.591282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb podName:6aa634f6-8f9c-4cd6-946d-bc018c3c20bc nodeName:}" failed. No retries permitted until 2026-10-31 02:54:44.591262798 +0000 UTC m=+10.169678745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6x7mb" (UniqueName: "kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb") pod "network-check-target-r76hd" (UID: "6aa634f6-8f9c-4cd6-946d-bc018c3c20bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:40.591809 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.591712 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:40.591809 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.591769 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret podName:566be63b-5b2b-4f4f-a40f-1e83e7837f74 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:44.591753434 +0000 UTC m=+10.170169396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret") pod "global-pull-secret-syncer-4pp9z" (UID: "566be63b-5b2b-4f4f-a40f-1e83e7837f74") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:40.916608 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.916525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:40.916782 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.916733 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:40.917285 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:40.917121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:40.917285 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:40.917235 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:41.045136 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:41.045012 2576 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="412085e90f40ed06da097d6f351f96c51838bc531ffed9447245c5dc9f0a4f9d" exitCode=0 Oct 31 02:54:41.045136 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:41.045047 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerDied","Data":"412085e90f40ed06da097d6f351f96c51838bc531ffed9447245c5dc9f0a4f9d"} Oct 31 02:54:41.076968 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:41.076478 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" podStartSLOduration=5.07646484 podStartE2EDuration="5.07646484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:40.060720387 +0000 UTC m=+5.639136355" watchObservedRunningTime="2026-10-31 02:54:41.07646484 +0000 UTC m=+6.654880809" Oct 31 02:54:41.916875 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:41.916423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:41.916875 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:41.916533 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:42.050249 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:42.049543 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sjjkf" event={"ID":"64b281bc-7922-46b5-8d33-2f1bfdebb8ec","Type":"ContainerStarted","Data":"41da253f69ddc81da3c50e31555a6d57b03252ef5c3134facfd9b516605c5fcd"} Oct 31 02:54:42.916568 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:42.916499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:42.916683 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:42.916612 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:42.918966 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:42.918943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:42.919070 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:42.919055 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:43.053286 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:43.053238 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8nv7d" event={"ID":"b408b194-9df2-4518-b247-0d9a83d80f3b","Type":"ContainerStarted","Data":"50cc3205d9ede86839a7fa175c1aaad77573e7aec30d586b42d821d82636d004"} Oct 31 02:54:43.056361 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:43.056330 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" event={"ID":"22caca16-dc54-4bac-8dff-b87cab1b8b60","Type":"ContainerStarted","Data":"c440c02df1ea2966d219848296c3e8bdf95005ccb98968bf1f9461cbd32e4675"} Oct 31 02:54:43.073271 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:43.073198 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sjjkf" podStartSLOduration=4.113558585 podStartE2EDuration="8.073178348s" totalImagesPullingTime="3.959619763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:42.070346356 +0000 UTC m=+7.648762318" watchObservedRunningTime="2026-10-31 02:54:43.073178348 +0000 UTC m=+8.651594319" Oct 31 02:54:43.916833 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:43.916746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:43.916987 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:43.916888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:44.482587 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.480728 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 02:54:44.525923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.525892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:44.526075 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.526029 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:44.526126 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.526095 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:52.526076039 +0000 UTC m=+18.104492004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:44.626268 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.626231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:44.626459 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.626309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:44.626543 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.626508 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:44.626543 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.626531 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:44.626644 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.626546 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6x7mb for pod openshift-network-diagnostics/network-check-target-r76hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:44.626644 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.626624 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb podName:6aa634f6-8f9c-4cd6-946d-bc018c3c20bc nodeName:}" failed. No retries permitted until 2026-10-31 02:54:52.626603316 +0000 UTC m=+18.205019264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6x7mb" (UniqueName: "kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb") pod "network-check-target-r76hd" (UID: "6aa634f6-8f9c-4cd6-946d-bc018c3c20bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:44.627068 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.627050 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:44.627131 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.627109 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret podName:566be63b-5b2b-4f4f-a40f-1e83e7837f74 nodeName:}" failed. No retries permitted until 2026-10-31 02:54:52.627093077 +0000 UTC m=+18.205509027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret") pod "global-pull-secret-syncer-4pp9z" (UID: "566be63b-5b2b-4f4f-a40f-1e83e7837f74") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:44.918583 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.918553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:44.918750 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.918680 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:44.920319 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.920295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:44.920428 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:44.920392 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:44.923457 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.923350 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T02:54:44.480748703Z","UUID":"4448872b-1baf-44d0-8823-7f7507f776be","Handler":null,"Name":"","Endpoint":""} Oct 31 02:54:44.926214 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.926192 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 02:54:44.926345 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:44.926224 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 02:54:45.064653 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:45.064571 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ltlq" event={"ID":"5e948ca8-1809-4eeb-b014-f563cb6fdb91","Type":"ContainerStarted","Data":"bf4e3dbbb362fc009bd257ebf9acd88e92cf7f4cc9967ee611271c87e54afae2"} Oct 31 02:54:45.068864 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:45.068829 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" event={"ID":"22caca16-dc54-4bac-8dff-b87cab1b8b60","Type":"ContainerStarted","Data":"2f2a1a519ba1c7c38e3a9351379aeb44b735ea54b1cf39252c56d25abfe6ee89"} Oct 31 02:54:45.071801 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:45.071773 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9rzxm" event={"ID":"9888db69-6183-403a-82cf-01ad3667c6f9","Type":"ContainerStarted","Data":"d7c37d8536039c6c5277ecb52ed076e7e36aae267905aae826c10494d1e35863"} Oct 31 02:54:45.085334 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:45.085257 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8nv7d" podStartSLOduration=4.971610292 podStartE2EDuration="10.08524131s" totalImagesPullingTime="5.113631018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:43.074232075 +0000 UTC m=+8.652648035" watchObservedRunningTime="2026-10-31 02:54:45.08524131 +0000 UTC m=+10.663657278" Oct 31 02:54:45.109111 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:45.109059 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2ltlq" podStartSLOduration=1.588858353 podStartE2EDuration="9.10903855s" totalImagesPullingTime="7.520180197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:45.086171151 +0000 UTC m=+10.664587117" watchObservedRunningTime="2026-10-31 02:54:45.10903855 +0000 UTC m=+10.687454518" Oct 31 02:54:45.916731 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:45.916553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:45.917044 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:45.916808 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:46.075691 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.075596 2576 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="023e23fad68c64b14e777f3d277976031f16510eb852983b7ed5d0097d8e885e" exitCode=0 Oct 31 02:54:46.075691 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.075648 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerDied","Data":"023e23fad68c64b14e777f3d277976031f16510eb852983b7ed5d0097d8e885e"} Oct 31 02:54:46.077973 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.077947 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" event={"ID":"22caca16-dc54-4bac-8dff-b87cab1b8b60","Type":"ContainerStarted","Data":"bbd0acdd2d1cf8c9a82f0e4908b57ed6d794ce9086103409d82824f795313b59"} Oct 31 02:54:46.103281 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.103230 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9rzxm" podStartSLOduration=3.092175372 podStartE2EDuration="10.103216722s" totalImagesPullingTime="7.01104135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:45.109440214 +0000 UTC m=+10.687856180" watchObservedRunningTime="2026-10-31 02:54:46.103216722 +0000 UTC m=+11.681632688" Oct 31 02:54:46.126161 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.126110 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" podStartSLOduration=2.155498068 podStartE2EDuration="10.126096422s" totalImagesPullingTime="7.970598354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:46.12585738 +0000 UTC m=+11.704273360" watchObservedRunningTime="2026-10-31 02:54:46.126096422 +0000 UTC m=+11.704512385" Oct 31 02:54:46.754092 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.754015 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:46.754646 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.754618 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:54:46.917406 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.917374 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:46.917831 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:46.917489 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:46.917831 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:46.917793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:46.917890 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:46.917857 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:47.082058 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:47.081970 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jpxqb" event={"ID":"70d72e18-bf2d-46e9-91ef-2374a0d41cf1","Type":"ContainerStarted","Data":"186d2c7513d9367732c7bae514f08b73fbee35c5c1fe6a09a80ba1c39493c635"} Oct 31 02:54:47.102757 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:47.102704 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jpxqb" podStartSLOduration=3.033933217 podStartE2EDuration="11.102689315s" totalImagesPullingTime="8.068756098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:47.102582077 +0000 UTC m=+12.680998041" watchObservedRunningTime="2026-10-31 02:54:47.102689315 +0000 UTC m=+12.681105285" Oct 31 02:54:47.916707 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:47.916676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:47.916863 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:47.916795 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:48.086764 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:48.086730 2576 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="c770db6c5aeb97accd028f6bb0aa4d0c01cd6c90b8b38a839585a3f75b3497fd" exitCode=0 Oct 31 02:54:48.087349 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:48.086820 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerDied","Data":"c770db6c5aeb97accd028f6bb0aa4d0c01cd6c90b8b38a839585a3f75b3497fd"} Oct 31 02:54:48.916437 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:48.916393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:48.916601 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:48.916448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:48.916601 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:48.916568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:48.916991 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:48.916964 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:49.093173 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:49.093142 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerStarted","Data":"0db4c2583cbbdde8b388f3f9919e90bd64d7a5047747a5a202d19d4e12e89800"} Oct 31 02:54:49.916702 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:49.916672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:49.916889 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:49.916830 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:50.098005 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:50.097978 2576 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="0db4c2583cbbdde8b388f3f9919e90bd64d7a5047747a5a202d19d4e12e89800" exitCode=0 Oct 31 02:54:50.098005 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:50.098020 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerDied","Data":"0db4c2583cbbdde8b388f3f9919e90bd64d7a5047747a5a202d19d4e12e89800"} Oct 31 02:54:50.916721 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:50.916428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:50.916889 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:50.916771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:50.917108 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:50.917080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:50.917230 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:50.917188 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:51.101849 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:51.101797 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vws7" event={"ID":"b6bfe2e7-b724-42de-acff-a45a1609c7d0","Type":"ContainerStarted","Data":"d94e095e5690e3603e86cd7c5892922f2a80870f17918c839b40a4d13e79fe61"} Oct 31 02:54:51.125567 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:51.123844 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7vws7" podStartSLOduration=2.798564835 podStartE2EDuration="16.123826762s" totalImagesPullingTime="13.325261927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:51.122744802 +0000 UTC m=+16.701160767" watchObservedRunningTime="2026-10-31 02:54:51.123826762 +0000 UTC m=+16.702242726" Oct 31 02:54:51.916868 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:51.916836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:51.917052 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:51.916965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:52.584888 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:52.584859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:52.585212 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.585002 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:52.585212 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.585071 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:08.585051703 +0000 UTC m=+34.163467667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:52.685455 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:52.685385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:52.685455 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:52.685449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:52.685678 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.685553 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:52.685678 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.685574 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:52.685678 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.685599 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:52.685678 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.685613 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6x7mb for pod openshift-network-diagnostics/network-check-target-r76hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:52.685678 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.685623 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret podName:566be63b-5b2b-4f4f-a40f-1e83e7837f74 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:08.685602576 +0000 UTC m=+34.264018517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret") pod "global-pull-secret-syncer-4pp9z" (UID: "566be63b-5b2b-4f4f-a40f-1e83e7837f74") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:52.685678 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.685663 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb podName:6aa634f6-8f9c-4cd6-946d-bc018c3c20bc nodeName:}" failed. No retries permitted until 2026-10-31 02:55:08.685646575 +0000 UTC m=+34.264062520 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6x7mb" (UniqueName: "kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb") pod "network-check-target-r76hd" (UID: "6aa634f6-8f9c-4cd6-946d-bc018c3c20bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:52.916899 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:52.916865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:52.916899 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:52.916884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:52.917091 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.916965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:52.917091 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:52.917082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:53.108673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:53.108606 2576 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="153134343ca5eaea75d9099509bbd77de62e51aa3cf9447035934f543d7aeeb0" exitCode=0 Oct 31 02:54:53.108673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:53.108656 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerDied","Data":"153134343ca5eaea75d9099509bbd77de62e51aa3cf9447035934f543d7aeeb0"} Oct 31 02:54:53.917288 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:53.917092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:53.917725 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:53.917386 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:54.113639 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.113605 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"2376edefd0f3e87a08de5872d71fedb1c2f660388429c23654b33d7e1b6855cf"} Oct 31 02:54:54.113791 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.113691 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"5da6e1673a4e6b4cff464c164f8faf58f730184c8c2ba2aa15c486f3a3ae7bd9"} Oct 31 02:54:54.113791 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.113742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"21dc30c37f063ceb810b70b441048c5628a1948c2ac540946cb8b13cfe20ea40"} Oct 31 02:54:54.113868 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.113798 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"b33a123cd467e75aa87f9acf35422317a8439956d95bcc5e388bc77240a253b5"} Oct 31 02:54:54.113868 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.113841 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"c21fd028a3fa59c610790c248640a5c777ef51b51cce59bb448d7c61ae680d03"} Oct 31 02:54:54.113952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.113872 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"8ec2515580e021b1b2befd607abb5254bd117894dbbc10cf0ec2d58e65cdefdb"} Oct 31 02:54:54.116876 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.116854 2576 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="5579075ff4753da6cf4c746d52b668bd2a851089131833e544f3120785a73973" exitCode=0 Oct 31 02:54:54.116987 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.116889 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerDied","Data":"5579075ff4753da6cf4c746d52b668bd2a851089131833e544f3120785a73973"} Oct 31 02:54:54.917757 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.917713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:54.918224 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:54.917849 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:54.918224 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:54.917899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:54.918224 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:54.917998 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:55.122107 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:55.122076 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" event={"ID":"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e","Type":"ContainerStarted","Data":"fe42dbdedad47e94c556ee213e634ec036742fcd035854403ba08833207b287f"} Oct 31 02:54:55.150951 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:55.150912 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qrb4n" podStartSLOduration=7.517589624 podStartE2EDuration="19.150899068s" totalImagesPullingTime="11.633309444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:55.148941914 +0000 UTC m=+20.727357876" watchObservedRunningTime="2026-10-31 02:54:55.150899068 +0000 UTC m=+20.729315060" Oct 31 02:54:55.916839 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:55.916770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:55.916953 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:55.916872 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:56.129204 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:56.129169 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"d84d1fe85fa187b3e1280dc9f8c518d24f195ccd737e8f6c1100021912fcbcf1"} Oct 31 02:54:56.916361 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:56.916333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:56.916361 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:56.916367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:56.916651 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:56.916449 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:56.916651 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:56.916590 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:57.916665 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:57.916631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:57.917369 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:57.916737 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:58.137265 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.137178 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" event={"ID":"dcf9a703-8b57-41ae-a801-cee51331a479","Type":"ContainerStarted","Data":"fe955958079c38f8b34bd10c207ccdfa83eb844ae2a65240d55c4858e23e0e8b"} Oct 31 02:54:58.137430 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.137414 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:58.137489 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.137477 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:58.152786 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.152764 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:58.152898 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.152852 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:54:58.170802 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.170765 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" podStartSLOduration=6.120110505 podStartE2EDuration="22.170754588s" totalImagesPullingTime="16.050644083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:58.17045428 +0000 UTC m=+23.748870266" watchObservedRunningTime="2026-10-31 02:54:58.170754588 +0000 UTC m=+23.749170551" Oct 31 02:54:58.916351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.916314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:58.916538 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:58.916427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:58.916538 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:58.916500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:58.916666 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:58.916609 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:54:59.139434 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.139406 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:54:59.549474 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.549447 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4pp9z"] Oct 31 02:54:59.549631 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.549555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:54:59.549703 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:59.549633 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:54:59.552136 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.552114 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qlrx"] Oct 31 02:54:59.552231 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.552189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:54:59.552293 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:59.552279 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:54:59.555437 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.555415 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r76hd"] Oct 31 02:54:59.555544 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:54:59.555500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:54:59.555607 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:54:59.555587 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:55:00.141606 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:00.141436 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:00.916758 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:00.916724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:00.916919 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:00.916725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:00.916959 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:00.916913 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:55:00.916959 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:00.916822 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:55:00.916959 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:00.916733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:00.917077 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:00.916998 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:55:02.916753 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:02.916662 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:02.916753 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:02.916688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:02.916753 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:02.916752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:02.917335 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:02.916832 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:55:02.917335 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:02.916918 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:55:02.917335 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:02.916975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:55:03.828405 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:03.828379 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:03.828624 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:03.828516 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:03.829160 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:03.829140 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:04.879881 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:04.879856 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:04.880325 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:04.880100 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:04.891795 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:04.891766 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:04.916566 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:04.916549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:04.916669 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:04.916554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:04.916669 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:04.916635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qlrx" podUID="111372d5-9b74-4bae-b96f-3671787ad5b3" Oct 31 02:55:04.916750 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:04.916707 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r76hd" podUID="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" Oct 31 02:55:04.916750 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:04.916745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:04.916811 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:04.916799 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pp9z" podUID="566be63b-5b2b-4f4f-a40f-1e83e7837f74" Oct 31 02:55:05.695346 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.695320 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeReady" Oct 31 02:55:05.695483 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.695428 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 02:55:05.733975 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.733943 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j5zbk"] Oct 31 02:55:05.793211 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.793189 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8mfvv"] Oct 31 02:55:05.793346 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.793327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:05.797571 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.797552 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 02:55:05.797722 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.797710 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 02:55:05.806633 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.806615 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9n9mr"] Oct 31 02:55:05.806756 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.806737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:05.818759 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.818738 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2"] Oct 31 02:55:05.818900 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.818883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:05.840890 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.840873 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l8j7j"] Oct 31 02:55:05.840999 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.840981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:05.846117 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.846098 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 02:55:05.863208 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.863166 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xth74"] Oct 31 02:55:05.863306 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.863286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:05.877491 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.877472 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75b45ffb98-wkq9k"] Oct 31 02:55:05.877640 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.877624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.882966 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.882946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c76\" (UniqueName: \"kubernetes.io/projected/c4987385-e27b-4220-92d4-2f61bdb1ebc8-kube-api-access-24c76\") pod \"kas-connection-checker-685895c7c9-j5zbk\" (UID: \"c4987385-e27b-4220-92d4-2f61bdb1ebc8\") " pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:05.883294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.882986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgj7m\" (UniqueName: \"kubernetes.io/projected/94b41492-1a35-4ede-adcd-65e4c41c6cc0-kube-api-access-mgj7m\") pod \"kas-connection-checker-685895c7c9-9n9mr\" (UID: \"94b41492-1a35-4ede-adcd-65e4c41c6cc0\") " pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:05.883294 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.883130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2tx\" (UniqueName: \"kubernetes.io/projected/e8f27649-b353-4181-bc11-3aa1ec714be3-kube-api-access-tp2tx\") pod \"kas-connection-checker-685895c7c9-8mfvv\" (UID: \"e8f27649-b353-4181-bc11-3aa1ec714be3\") " pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:05.896209 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.896193 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk"] Oct 31 02:55:05.896333 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.896319 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.904402 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904380 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 02:55:05.904491 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904419 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 02:55:05.904491 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904429 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 02:55:05.904491 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904453 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 02:55:05.904491 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904487 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:05.904667 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904564 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 02:55:05.904667 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904617 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:05.904769 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.904711 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 02:55:05.910334 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.910317 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj"] Oct 31 02:55:05.910467 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.910453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:05.911517 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.911497 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 02:55:05.920512 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.920495 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 02:55:05.921478 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.921464 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 02:55:05.924424 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.924402 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 02:55:05.934472 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.934455 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5k4sc"] Oct 31 02:55:05.934590 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.934577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:05.946320 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946292 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 02:55:05.946428 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946414 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 02:55:05.946584 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946563 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 02:55:05.946687 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946566 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sc997\"" Oct 31 02:55:05.946687 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946638 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 02:55:05.946687 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946678 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mzmdp\"" Oct 31 02:55:05.946882 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946735 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bfddz\"" Oct 31 02:55:05.946882 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.946740 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 02:55:05.958644 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.958628 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66695d6bd7-5pk65"] Oct 31 02:55:05.958757 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.958744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:05.965190 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.965176 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 02:55:05.978453 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.978429 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46"] Oct 31 02:55:05.978523 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.978508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:05.983456 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-serving-cert\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:05.983562 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:05.983722 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c76\" (UniqueName: \"kubernetes.io/projected/c4987385-e27b-4220-92d4-2f61bdb1ebc8-kube-api-access-24c76\") pod \"kas-connection-checker-685895c7c9-j5zbk\" (UID: \"c4987385-e27b-4220-92d4-2f61bdb1ebc8\") " pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:05.983722 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.983722 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-tmp\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.983722 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:05.983952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgj7m\" (UniqueName: \"kubernetes.io/projected/94b41492-1a35-4ede-adcd-65e4c41c6cc0-kube-api-access-mgj7m\") pod \"kas-connection-checker-685895c7c9-9n9mr\" (UID: \"94b41492-1a35-4ede-adcd-65e4c41c6cc0\") " pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:05.983952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.983952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.983952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2k9\" (UniqueName: \"kubernetes.io/projected/e7194787-0fa9-43a4-a749-c6aac9433cfc-kube-api-access-5z2k9\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.983952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-config\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:05.983952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.983966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-snapshots\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7194787-0fa9-43a4-a749-c6aac9433cfc-serving-cert\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-service-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qbk\" (UniqueName: \"kubernetes.io/projected/05ce07aa-7ab8-45da-85c1-f036dc52c6b8-kube-api-access-28qbk\") pod \"volume-data-source-validator-58c9f5f46d-xgpqk\" (UID: \"05ce07aa-7ab8-45da-85c1-f036dc52c6b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:05.984272 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdbc\" (UniqueName: \"kubernetes.io/projected/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-kube-api-access-gsdbc\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:05.984607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvcmb\" (UniqueName: \"kubernetes.io/projected/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-kube-api-access-pvcmb\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:05.984607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.984607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.984607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2tx\" (UniqueName: \"kubernetes.io/projected/e8f27649-b353-4181-bc11-3aa1ec714be3-kube-api-access-tp2tx\") pod \"kas-connection-checker-685895c7c9-8mfvv\" (UID: \"e8f27649-b353-4181-bc11-3aa1ec714be3\") " pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:05.984607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.984432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:05.998311 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.998289 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 02:55:05.998412 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.998315 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 02:55:05.998412 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.998342 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:05.998412 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.998377 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 02:55:05.998516 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.998419 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 02:55:05.998516 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:05.998451 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:06.001523 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.001506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgj7m\" (UniqueName: \"kubernetes.io/projected/94b41492-1a35-4ede-adcd-65e4c41c6cc0-kube-api-access-mgj7m\") pod \"kas-connection-checker-685895c7c9-9n9mr\" (UID: \"94b41492-1a35-4ede-adcd-65e4c41c6cc0\") " pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:06.001607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.001508 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2tx\" (UniqueName: \"kubernetes.io/projected/e8f27649-b353-4181-bc11-3aa1ec714be3-kube-api-access-tp2tx\") pod \"kas-connection-checker-685895c7c9-8mfvv\" (UID: \"e8f27649-b353-4181-bc11-3aa1ec714be3\") " pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:06.001607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.001506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c76\" (UniqueName: \"kubernetes.io/projected/c4987385-e27b-4220-92d4-2f61bdb1ebc8-kube-api-access-24c76\") pod \"kas-connection-checker-685895c7c9-j5zbk\" (UID: \"c4987385-e27b-4220-92d4-2f61bdb1ebc8\") " pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:06.002741 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.002719 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp"] Oct 31 02:55:06.002877 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.002863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.022131 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.022115 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf"] Oct 31 02:55:06.022229 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.022216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.045932 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.045909 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw"] Oct 31 02:55:06.046104 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.046381 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046210 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 02:55:06.046381 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046319 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 02:55:06.046381 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046335 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 02:55:06.046550 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046383 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 02:55:06.046550 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046386 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 02:55:06.046550 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046405 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hv8gl\"" Oct 31 02:55:06.046550 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.046474 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mqqj2\"" Oct 31 02:55:06.070423 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.070400 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j5zbk"] Oct 31 02:55:06.070535 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.070432 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pwpgn"] Oct 31 02:55:06.070535 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.070503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:06.083044 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.083027 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ss5vz"] Oct 31 02:55:06.083158 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.083143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.084879 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.084727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpnn\" (UniqueName: \"kubernetes.io/projected/e95878f0-c838-42c0-a546-55f075727d45-kube-api-access-7qpnn\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.085030 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.084892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-stats-auth\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.085030 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.084945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-serving-cert\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.085138 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.085138 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:06.085303 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.085303 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.085303 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2k9\" (UniqueName: \"kubernetes.io/projected/e7194787-0fa9-43a4-a749-c6aac9433cfc-kube-api-access-5z2k9\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.085303 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-default-certificate\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.085488 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cvz\" (UniqueName: \"kubernetes.io/projected/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-kube-api-access-p4cvz\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.085488 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95878f0-c838-42c0-a546-55f075727d45-serving-cert\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.085488 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.085488 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.085375 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:06.085626 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.085626 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.085626 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.085626 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.085618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.085908 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.085891 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.585864797 +0000 UTC m=+32.164280739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:06.086403 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.086506 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-snapshots\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.086506 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.086506 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.086644 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086533 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.086644 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.086733 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-service-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.086789 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvcmb\" (UniqueName: \"kubernetes.io/projected/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-kube-api-access-pvcmb\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:06.086860 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.086952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:06.087011 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.087011 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.086983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.087109 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e95878f0-c838-42c0-a546-55f075727d45-config\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.087109 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:06.087109 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.087109 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.087105 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.087121 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2nn\" (UniqueName: \"kubernetes.io/projected/aca0a795-839c-4055-b009-793e1ffc4d58-kube-api-access-lf2nn\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087025 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-snapshots\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.087184 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.587164114 +0000 UTC m=+32.165580069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087194 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-service-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-tmp\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.087351 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.087673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.087673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.087673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.087673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.087673 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087619 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-tmp\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087735 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-config\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7194787-0fa9-43a4-a749-c6aac9433cfc-serving-cert\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eda75e-e883-4929-8278-fd49da0f5b27-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.087923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087908 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.088326 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qbk\" (UniqueName: \"kubernetes.io/projected/05ce07aa-7ab8-45da-85c1-f036dc52c6b8-kube-api-access-28qbk\") pod \"volume-data-source-validator-58c9f5f46d-xgpqk\" (UID: \"05ce07aa-7ab8-45da-85c1-f036dc52c6b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:06.088326 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.087964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdbc\" (UniqueName: \"kubernetes.io/projected/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-kube-api-access-gsdbc\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.088790 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.088762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-serving-cert\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.088874 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.088820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-config\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.088988 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.088964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.089712 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.089685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.090175 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.090152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.090313 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.090295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7194787-0fa9-43a4-a749-c6aac9433cfc-serving-cert\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.097160 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.097141 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098411 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098431 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098498 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098627 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098627 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098639 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098687 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098707 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098749 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:06.098774 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.098765 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:06.099265 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.099226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2k9\" (UniqueName: \"kubernetes.io/projected/e7194787-0fa9-43a4-a749-c6aac9433cfc-kube-api-access-5z2k9\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.099370 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.099354 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:06.099434 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.099399 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:06.099546 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.099530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.099714 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.099700 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvcmb\" (UniqueName: \"kubernetes.io/projected/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-kube-api-access-pvcmb\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:06.104361 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.104337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdbc\" (UniqueName: \"kubernetes.io/projected/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-kube-api-access-gsdbc\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.109193 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.109171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qbk\" (UniqueName: \"kubernetes.io/projected/05ce07aa-7ab8-45da-85c1-f036dc52c6b8-kube-api-access-28qbk\") pod \"volume-data-source-validator-58c9f5f46d-xgpqk\" (UID: \"05ce07aa-7ab8-45da-85c1-f036dc52c6b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:06.111086 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111066 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8mfvv"] Oct 31 02:55:06.111160 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111101 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk"] Oct 31 02:55:06.111160 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111122 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9n9mr"] Oct 31 02:55:06.111160 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111138 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw"] Oct 31 02:55:06.111160 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111150 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111164 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111185 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75b45ffb98-wkq9k"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111198 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111211 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111289 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwpgn"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111313 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l8j7j"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111331 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111348 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ss5vz"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111368 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xth74"] Oct 31 02:55:06.111386 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111384 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5k4sc"] Oct 31 02:55:06.111732 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.111403 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66695d6bd7-5pk65"] Oct 31 02:55:06.119449 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.119431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:06.135994 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.135979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:06.146426 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146407 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 02:55:06.146505 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146448 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 02:55:06.146505 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146461 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 02:55:06.146505 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146496 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gxlr7\"" Oct 31 02:55:06.146644 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146630 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x8dcs\"" Oct 31 02:55:06.146719 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146694 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jsfq8\"" Oct 31 02:55:06.146755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.146708 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jp47q\"" Oct 31 02:55:06.153850 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.153825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:06.183890 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.183861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:06.188928 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.188904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.189026 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.188934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.189026 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.188954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs5b\" (UniqueName: \"kubernetes.io/projected/fb123961-0503-4503-b9d4-6790d658e12e-kube-api-access-vcs5b\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.189026 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.188974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-default-certificate\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.189026 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.188993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cvz\" (UniqueName: \"kubernetes.io/projected/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-kube-api-access-p4cvz\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.189026 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95878f0-c838-42c0-a546-55f075727d45-serving-cert\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.189026 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9t8c\" (UniqueName: \"kubernetes.io/projected/616e84dc-1530-44cd-b4b3-752e5635a0ea-kube-api-access-t9t8c\") pod \"network-check-source-6f78d59f99-zp7jw\" (UID: \"616e84dc-1530-44cd-b4b3-752e5635a0ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e95878f0-c838-42c0-a546-55f075727d45-config\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2nn\" (UniqueName: \"kubernetes.io/projected/aca0a795-839c-4055-b009-793e1ffc4d58-kube-api-access-lf2nn\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8ch\" (UniqueName: \"kubernetes.io/projected/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-kube-api-access-7r8ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189309 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8a54-63a4-408c-ae91-e21314a80325-tmp-dir\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75tc\" (UniqueName: \"kubernetes.io/projected/b3be8a54-63a4-408c-ae91-e21314a80325-kube-api-access-r75tc\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eda75e-e883-4929-8278-fd49da0f5b27-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb123961-0503-4503-b9d4-6790d658e12e-cert\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpnn\" (UniqueName: \"kubernetes.io/projected/e95878f0-c838-42c0-a546-55f075727d45-kube-api-access-7qpnn\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-stats-auth\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.189514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3be8a54-63a4-408c-ae91-e21314a80325-config-volume\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.189652 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:06.189923 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.189700 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.689687352 +0000 UTC m=+32.268103310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:06.190530 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.190425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.190915 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.190898 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:06.190915 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.190914 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:06.191095 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.190953 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.690939314 +0000 UTC m=+32.269355268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:06.191095 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.191011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.691000972 +0000 UTC m=+32.269416933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:06.191993 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.191371 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:06.191993 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.191427 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.691410607 +0000 UTC m=+32.269826751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:06.192361 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.192342 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:06.192443 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.192384 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.692373401 +0000 UTC m=+32.270789379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:06.201498 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.201378 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 02:55:06.201498 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.201450 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.204105 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.204375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e95878f0-c838-42c0-a546-55f075727d45-config\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.204459 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eda75e-e883-4929-8278-fd49da0f5b27-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.204491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.204950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95878f0-c838-42c0-a546-55f075727d45-serving-cert\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.205133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-default-certificate\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.205298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.205489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.205755 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.205719 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.207607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.207307 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-stats-auth\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.207607 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.207575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.210670 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.210631 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.211179 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.211153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.211496 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.211456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2nn\" (UniqueName: \"kubernetes.io/projected/aca0a795-839c-4055-b009-793e1ffc4d58-kube-api-access-lf2nn\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.211893 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.211872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpnn\" (UniqueName: \"kubernetes.io/projected/e95878f0-c838-42c0-a546-55f075727d45-kube-api-access-7qpnn\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.212202 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.212183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cvz\" (UniqueName: \"kubernetes.io/projected/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-kube-api-access-p4cvz\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.251295 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.249395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:06.290593 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9t8c\" (UniqueName: \"kubernetes.io/projected/616e84dc-1530-44cd-b4b3-752e5635a0ea-kube-api-access-t9t8c\") pod \"network-check-source-6f78d59f99-zp7jw\" (UID: \"616e84dc-1530-44cd-b4b3-752e5635a0ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:06.290692 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.290692 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.290692 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.290839 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8ch\" (UniqueName: \"kubernetes.io/projected/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-kube-api-access-7r8ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.290839 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8a54-63a4-408c-ae91-e21314a80325-tmp-dir\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.290839 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75tc\" (UniqueName: \"kubernetes.io/projected/b3be8a54-63a4-408c-ae91-e21314a80325-kube-api-access-r75tc\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.290839 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb123961-0503-4503-b9d4-6790d658e12e-cert\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.290839 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3be8a54-63a4-408c-ae91-e21314a80325-config-volume\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.291041 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.290852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs5b\" (UniqueName: \"kubernetes.io/projected/fb123961-0503-4503-b9d4-6790d658e12e-kube-api-access-vcs5b\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.291462 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.291440 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8a54-63a4-408c-ae91-e21314a80325-tmp-dir\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.292320 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.292040 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:06.292320 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.292113 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:06.792095889 +0000 UTC m=+32.370511844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:06.292489 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.292438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3be8a54-63a4-408c-ae91-e21314a80325-config-volume\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.292762 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.292727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.297526 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.297480 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.301792 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.301749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb123961-0503-4503-b9d4-6790d658e12e-cert\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.309006 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.305945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs5b\" (UniqueName: \"kubernetes.io/projected/fb123961-0503-4503-b9d4-6790d658e12e-kube-api-access-vcs5b\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.309263 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.309187 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9t8c\" (UniqueName: \"kubernetes.io/projected/616e84dc-1530-44cd-b4b3-752e5635a0ea-kube-api-access-t9t8c\") pod \"network-check-source-6f78d59f99-zp7jw\" (UID: \"616e84dc-1530-44cd-b4b3-752e5635a0ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:06.314583 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.313575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75tc\" (UniqueName: \"kubernetes.io/projected/b3be8a54-63a4-408c-ae91-e21314a80325-kube-api-access-r75tc\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.315871 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.315820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8ch\" (UniqueName: \"kubernetes.io/projected/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-kube-api-access-7r8ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.337460 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.337431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:06.396429 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.396357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:06.421555 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.421529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:06.471507 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.471304 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8mfvv"] Oct 31 02:55:06.474487 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.473856 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:06.474487 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.474111 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9n9mr"] Oct 31 02:55:06.474487 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.474438 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f27649_b353_4181_bc11_3aa1ec714be3.slice/crio-c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24 WatchSource:0}: Error finding container c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24: Status 404 returned error can't find the container with id c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24 Oct 31 02:55:06.475602 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.475579 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j5zbk"] Oct 31 02:55:06.477400 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.477378 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b41492_1a35_4ede_adcd_65e4c41c6cc0.slice/crio-9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a WatchSource:0}: Error finding container 9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a: Status 404 returned error can't find the container with id 9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a Oct 31 02:55:06.478092 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.478072 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4987385_e27b_4220_92d4_2f61bdb1ebc8.slice/crio-e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d WatchSource:0}: Error finding container e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d: Status 404 returned error can't find the container with id e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d Oct 31 02:55:06.530075 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.530048 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l8j7j"] Oct 31 02:55:06.560292 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.560239 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xth74"] Oct 31 02:55:06.587045 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.584813 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk"] Oct 31 02:55:06.592633 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.591951 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ce07aa_7ab8_45da_85c1_f036dc52c6b8.slice/crio-81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e WatchSource:0}: Error finding container 81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e: Status 404 returned error can't find the container with id 81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.594624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.594760 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.594776 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.594823 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.594832 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.594806848 +0000 UTC m=+33.173222806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.594759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:06.594928 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.594857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.594846833 +0000 UTC m=+33.173262789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:06.633095 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.632708 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46"] Oct 31 02:55:06.635955 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.635926 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95878f0_c838_42c0_a546_55f075727d45.slice/crio-9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1 WatchSource:0}: Error finding container 9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1: Status 404 returned error can't find the container with id 9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1 Oct 31 02:55:06.676547 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.676528 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf"] Oct 31 02:55:06.678974 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.678951 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eb238c_4f7d_47ec_8e6a_3fe59bdbdea8.slice/crio-0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850 WatchSource:0}: Error finding container 0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850: Status 404 returned error can't find the container with id 0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850 Oct 31 02:55:06.695947 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.695925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.696042 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.695961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:06.696042 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.696001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:06.696042 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.696030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.696048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696073 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696128 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.696108429 +0000 UTC m=+33.274524388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696134 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696138 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696136 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696157 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:06.696194 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696185 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.696170404 +0000 UTC m=+33.274586368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:06.696625 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696204 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.696191192 +0000 UTC m=+33.274607135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:06.696625 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696220 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.69621199 +0000 UTC m=+33.274627939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:06.696625 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.696233 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.696227342 +0000 UTC m=+33.274643283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:06.715608 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.715590 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw"] Oct 31 02:55:06.717531 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.717508 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616e84dc_1530_44cd_b4b3_752e5635a0ea.slice/crio-518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf WatchSource:0}: Error finding container 518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf: Status 404 returned error can't find the container with id 518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf Oct 31 02:55:06.791761 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.791660 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ss5vz"] Oct 31 02:55:06.794108 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:06.794087 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb123961_0503_4503_b9d4_6790d658e12e.slice/crio-ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556 WatchSource:0}: Error finding container ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556: Status 404 returned error can't find the container with id ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556 Oct 31 02:55:06.796610 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.796550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:06.796684 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.796669 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:06.796733 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:06.796726 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:07.796710853 +0000 UTC m=+33.375126808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:06.917362 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.917282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:06.917362 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.917315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:06.918282 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.917492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:06.921994 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.921972 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 02:55:06.922120 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.922058 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 02:55:06.922769 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.922743 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7qc7w\"" Oct 31 02:55:06.922892 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.922877 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q5ns6\"" Oct 31 02:55:06.922946 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:06.922904 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkjm6\"" Oct 31 02:55:07.159954 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.159921 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ss5vz" event={"ID":"fb123961-0503-4503-b9d4-6790d658e12e","Type":"ContainerStarted","Data":"ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556"} Oct 31 02:55:07.162287 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.162234 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" event={"ID":"e8f27649-b353-4181-bc11-3aa1ec714be3","Type":"ContainerStarted","Data":"ef85f76267dd216a9f5f7ee0402554d7f1c7892596dab6b588a802845990a973"} Oct 31 02:55:07.162396 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.162309 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" event={"ID":"e8f27649-b353-4181-bc11-3aa1ec714be3","Type":"ContainerStarted","Data":"c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24"} Oct 31 02:55:07.163568 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.163536 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" event={"ID":"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8","Type":"ContainerStarted","Data":"0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850"} Oct 31 02:55:07.164926 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.164884 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" event={"ID":"e95878f0-c838-42c0-a546-55f075727d45","Type":"ContainerStarted","Data":"9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1"} Oct 31 02:55:07.166206 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.166183 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" event={"ID":"05ce07aa-7ab8-45da-85c1-f036dc52c6b8","Type":"ContainerStarted","Data":"81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e"} Oct 31 02:55:07.167743 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.167694 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xth74" event={"ID":"e7194787-0fa9-43a4-a749-c6aac9433cfc","Type":"ContainerStarted","Data":"468fbf8a27df2d7b14e59052b97ae41f195509544f67e39e96a56f849fa05f28"} Oct 31 02:55:07.169687 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.169663 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" event={"ID":"94b41492-1a35-4ede-adcd-65e4c41c6cc0","Type":"ContainerStarted","Data":"0c42d79d7626d9e1c94cdc08ab786b4b9e0ab18a2a622ed0dd9ad67c8a96589d"} Oct 31 02:55:07.169796 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.169707 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" event={"ID":"94b41492-1a35-4ede-adcd-65e4c41c6cc0","Type":"ContainerStarted","Data":"9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a"} Oct 31 02:55:07.171519 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.171497 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" event={"ID":"c4987385-e27b-4220-92d4-2f61bdb1ebc8","Type":"ContainerStarted","Data":"fd1635bad93f50b0a6b13920cd4d88b920fae9cbdae01ace32e4365e44800a7a"} Oct 31 02:55:07.171622 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.171538 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" event={"ID":"c4987385-e27b-4220-92d4-2f61bdb1ebc8","Type":"ContainerStarted","Data":"e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d"} Oct 31 02:55:07.172826 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.172802 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" event={"ID":"616e84dc-1530-44cd-b4b3-752e5635a0ea","Type":"ContainerStarted","Data":"518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf"} Oct 31 02:55:07.174044 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.174026 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" event={"ID":"ec216fff-d06e-4a0a-b35e-73b3a2d513a8","Type":"ContainerStarted","Data":"1f536b103ce85f26096da3b73015998c0ef840587ce49fd3f77dec3aa6f8ab94"} Oct 31 02:55:07.184574 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.184537 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" podStartSLOduration=393.184524763 podStartE2EDuration="6m33.184524763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:07.184012649 +0000 UTC m=+32.762428612" watchObservedRunningTime="2026-10-31 02:55:07.184524763 +0000 UTC m=+32.762940725" Oct 31 02:55:07.205273 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.205222 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" podStartSLOduration=393.205211233 podStartE2EDuration="6m33.205211233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:07.203528819 +0000 UTC m=+32.781944781" watchObservedRunningTime="2026-10-31 02:55:07.205211233 +0000 UTC m=+32.783627190" Oct 31 02:55:07.606320 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.606234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:07.606506 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.606378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:07.606506 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.606387 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:07.606624 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.606541 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.606522152 +0000 UTC m=+35.184938093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:07.607015 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.606624 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:07.607015 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.606636 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:07.607015 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.606681 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.606668703 +0000 UTC m=+35.185084644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:07.707009 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.706971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:07.707207 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.707040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:07.707207 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.707113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:07.707207 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.707165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:07.707207 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707187 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:07.707207 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.707193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:07.707486 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707272 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.707235549 +0000 UTC m=+35.285651513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:07.707486 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707322 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.707311576 +0000 UTC m=+35.285727523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:07.707486 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707385 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:07.707486 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707419 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.707407375 +0000 UTC m=+35.285823331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:07.707486 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707474 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:07.707731 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707504 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.707493887 +0000 UTC m=+35.285909828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:07.707731 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707560 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:07.707731 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707569 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:07.707731 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.707598 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.707589295 +0000 UTC m=+35.286005235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:07.808152 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:07.808114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:07.808369 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.808313 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:07.808432 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:07.808374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:09.808355832 +0000 UTC m=+35.386771773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:08.616285 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.616225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:08.617071 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:08.616486 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:08.617071 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:08.616561 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:40.616540522 +0000 UTC m=+66.194956469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : secret "metrics-daemon-secret" not found Oct 31 02:55:08.717660 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.717628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:08.717822 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.717766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:08.738717 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.738689 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:08.740852 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.740820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:08.770018 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.769982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:08.790566 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:08.790540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:09.068086 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.068040 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" podStartSLOduration=395.068026527 podStartE2EDuration="6m35.068026527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:07.224858857 +0000 UTC m=+32.803274820" watchObservedRunningTime="2026-10-31 02:55:09.068026527 +0000 UTC m=+34.646442490" Oct 31 02:55:09.068886 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.068823 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4pp9z"] Oct 31 02:55:09.122077 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.121503 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r76hd"] Oct 31 02:55:09.138656 ip-10-0-130-12 kubenswrapper[2576]: W1031 02:55:09.138625 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa634f6_8f9c_4cd6_946d_bc018c3c20bc.slice/crio-e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe WatchSource:0}: Error finding container e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe: Status 404 returned error can't find the container with id e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe Oct 31 02:55:09.184631 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.184602 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r76hd" event={"ID":"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc","Type":"ContainerStarted","Data":"e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe"} Oct 31 02:55:09.186622 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.186069 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4pp9z" event={"ID":"566be63b-5b2b-4f4f-a40f-1e83e7837f74","Type":"ContainerStarted","Data":"1a04461bfea6f53e187034091ed891d992dc1a3897e00b3453c980b12b92678b"} Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.633096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.633205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.633434 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.633449 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.633504 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.633487359 +0000 UTC m=+39.211903306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.633583 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:09.633947 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.633615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.633605059 +0000 UTC m=+39.212021005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.739397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.739435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.739472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.739501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.739516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739635 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.739621212 +0000 UTC m=+39.318037153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739695 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739716 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.73971011 +0000 UTC m=+39.318126051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739749 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739776 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.739767339 +0000 UTC m=+39.318183279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739829 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.739850119 +0000 UTC m=+39.318266060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739924 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739935 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:09.740357 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.739963 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.739953503 +0000 UTC m=+39.318369451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:09.841522 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:09.840848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:09.841522 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.841132 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:09.841522 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:09.841194 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.841176735 +0000 UTC m=+39.419592693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:10.195296 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:10.194656 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" event={"ID":"05ce07aa-7ab8-45da-85c1-f036dc52c6b8","Type":"ContainerStarted","Data":"467d025f4ef6b9639945e23dae8defaf3ecccce024b2baa10ade398d15fdd5da"} Oct 31 02:55:10.216763 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:10.215441 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" podStartSLOduration=378.744005079 podStartE2EDuration="6m21.215422072s" totalImagesPullingTime="2.471416993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:10.214772321 +0000 UTC m=+35.793188301" watchObservedRunningTime="2026-10-31 02:55:10.215422072 +0000 UTC m=+35.793838039" Oct 31 02:55:11.205130 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:11.204754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/0.log" Oct 31 02:55:11.205130 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:11.204805 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" containerID="15dae3965f70b0c3073cf50352fce8f612f0df04d5d1d6bfa5bbf96844ae47e5" exitCode=255 Oct 31 02:55:11.205130 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:11.204917 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" event={"ID":"ec216fff-d06e-4a0a-b35e-73b3a2d513a8","Type":"ContainerDied","Data":"15dae3965f70b0c3073cf50352fce8f612f0df04d5d1d6bfa5bbf96844ae47e5"} Oct 31 02:55:11.205130 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:11.205083 2576 scope.go:122] "RemoveContainer" containerID="15dae3965f70b0c3073cf50352fce8f612f0df04d5d1d6bfa5bbf96844ae47e5" Oct 31 02:55:12.227369 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.224231 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xth74" event={"ID":"e7194787-0fa9-43a4-a749-c6aac9433cfc","Type":"ContainerStarted","Data":"dfba916d4f8584000e1b335f54b8294243be242bf9f5357020641cfcafa64af4"} Oct 31 02:55:12.229537 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.229452 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/1.log" Oct 31 02:55:12.231197 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.230104 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/0.log" Oct 31 02:55:12.231197 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.230139 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" exitCode=255 Oct 31 02:55:12.231197 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.230218 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" event={"ID":"ec216fff-d06e-4a0a-b35e-73b3a2d513a8","Type":"ContainerDied","Data":"8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7"} Oct 31 02:55:12.231197 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.230291 2576 scope.go:122] "RemoveContainer" containerID="15dae3965f70b0c3073cf50352fce8f612f0df04d5d1d6bfa5bbf96844ae47e5" Oct 31 02:55:12.231197 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.230659 2576 scope.go:122] "RemoveContainer" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" Oct 31 02:55:12.231197 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:12.230840 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:12.236297 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.235849 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" event={"ID":"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8","Type":"ContainerStarted","Data":"e38e12c9832fefe2d5d24a47cb0abd4e99e758d1540da31c35af53a538f22b3c"} Oct 31 02:55:12.248346 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.247514 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xth74" podStartSLOduration=380.074615095 podStartE2EDuration="6m25.24749678s" totalImagesPullingTime="5.172881685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:12.2463314 +0000 UTC m=+37.824747363" watchObservedRunningTime="2026-10-31 02:55:12.24749678 +0000 UTC m=+37.825912744" Oct 31 02:55:12.301102 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.300016 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" podStartSLOduration=360.227860729 podStartE2EDuration="6m5.29999715s" totalImagesPullingTime="5.072136421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:12.298884284 +0000 UTC m=+37.877300247" watchObservedRunningTime="2026-10-31 02:55:12.29999715 +0000 UTC m=+37.878413121" Oct 31 02:55:12.670712 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.670677 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rsvkk"] Oct 31 02:55:12.688137 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.688109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.688295 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.688211 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rsvkk"] Oct 31 02:55:12.692553 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.692329 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 02:55:12.692553 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.692493 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 02:55:12.692737 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.692592 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sgkvt\"" Oct 31 02:55:12.781617 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.781585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84351d2c-2829-4e33-afa0-0f3f813dd4fa-data-volume\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.781617 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.781622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.781811 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.781710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhf8\" (UniqueName: \"kubernetes.io/projected/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-api-access-qqhf8\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.781811 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.781782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84351d2c-2829-4e33-afa0-0f3f813dd4fa-crio-socket\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.781811 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.781800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.882550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhf8\" (UniqueName: \"kubernetes.io/projected/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-api-access-qqhf8\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.882659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84351d2c-2829-4e33-afa0-0f3f813dd4fa-crio-socket\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.882686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.882723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84351d2c-2829-4e33-afa0-0f3f813dd4fa-data-volume\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.882749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:12.882949 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:12.882998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.382985821 +0000 UTC m=+38.961401775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.883427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84351d2c-2829-4e33-afa0-0f3f813dd4fa-crio-socket\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:12.883479 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:12.883520 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.383506445 +0000 UTC m=+38.961922389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : configmap "kube-rbac-proxy" not found Oct 31 02:55:12.883818 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.883762 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84351d2c-2829-4e33-afa0-0f3f813dd4fa-data-volume\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:12.897391 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:12.897363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhf8\" (UniqueName: \"kubernetes.io/projected/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-api-access-qqhf8\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:13.241967 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.241936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/1.log" Oct 31 02:55:13.242745 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.242730 2576 scope.go:122] "RemoveContainer" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" Oct 31 02:55:13.242911 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.242879 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:13.387604 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.387568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:13.387776 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.387640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:13.387776 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.387701 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:13.387878 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.387787 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:14.387765961 +0000 UTC m=+39.966181919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : configmap "kube-rbac-proxy" not found Oct 31 02:55:13.387878 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.387807 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:13.387950 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.387888 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:14.387868846 +0000 UTC m=+39.966284801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:13.690611 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.690562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:13.690806 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.690666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:13.690806 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.690774 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:13.690923 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.690819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.690806605 +0000 UTC m=+47.269222546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:13.691176 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.691156 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:13.691176 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.691173 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:13.691344 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.691207 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.691196502 +0000 UTC m=+47.269612442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:13.791295 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.791226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:13.791454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.791331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:13.791454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.791359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:13.791454 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.791450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:13.791612 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.791499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:13.791661 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.791648 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:13.791720 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.791708 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.791690124 +0000 UTC m=+47.370106072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:13.792125 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792111 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:13.792175 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792158 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.792145762 +0000 UTC m=+47.370561707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:13.792229 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792214 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:13.792229 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792225 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:13.792347 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792334 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:13.792449 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792439 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.792244506 +0000 UTC m=+47.370660451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:13.792506 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792457 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.792449245 +0000 UTC m=+47.370865187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:13.792506 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.792485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.792463385 +0000 UTC m=+47.370879331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:13.891952 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:13.891923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:13.892293 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.892274 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:13.892356 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:13.892330 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.892316934 +0000 UTC m=+47.470732889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:14.247849 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.247811 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ss5vz" event={"ID":"fb123961-0503-4503-b9d4-6790d658e12e","Type":"ContainerStarted","Data":"67811196cee88b5834a223672e0044f62b377293c3e32d4aed35cd277e635ebf"} Oct 31 02:55:14.249747 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.249708 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" event={"ID":"e95878f0-c838-42c0-a546-55f075727d45","Type":"ContainerStarted","Data":"f69f3dba55a7674deb6afe025f01c2cfddbf8d6cd91335ddd4af15bcbc459b07"} Oct 31 02:55:14.251467 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.251440 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r76hd" event={"ID":"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc","Type":"ContainerStarted","Data":"3f523e035e32f4602b5ff93c964c48c2d42094e53ad44777df984e84557ec0b3"} Oct 31 02:55:14.251614 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.251548 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:14.253222 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.253200 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" event={"ID":"616e84dc-1530-44cd-b4b3-752e5635a0ea","Type":"ContainerStarted","Data":"e1b05ab09a42c6b36e29820b5821e9dc06dacdca20e4b742aac2685869a24f95"} Oct 31 02:55:14.267978 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.267930 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ss5vz" podStartSLOduration=2.516079952 podStartE2EDuration="9.26791273s" totalImagesPullingTime="6.751832778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:14.266689126 +0000 UTC m=+39.845105091" watchObservedRunningTime="2026-10-31 02:55:14.26791273 +0000 UTC m=+39.846328694" Oct 31 02:55:14.286165 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.286110 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" podStartSLOduration=384.377721835 podStartE2EDuration="6m31.286094523s" totalImagesPullingTime="6.908372688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:14.284924924 +0000 UTC m=+39.863340888" watchObservedRunningTime="2026-10-31 02:55:14.286094523 +0000 UTC m=+39.864510487" Oct 31 02:55:14.304425 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.304381 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" podStartSLOduration=360.457787117 podStartE2EDuration="6m7.304365607s" totalImagesPullingTime="6.84657849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:14.303614496 +0000 UTC m=+39.882030459" watchObservedRunningTime="2026-10-31 02:55:14.304365607 +0000 UTC m=+39.882781570" Oct 31 02:55:14.321898 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.321843 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r76hd" podStartSLOduration=33.631463641 podStartE2EDuration="38.321826934s" totalImagesPullingTime="4.690363293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:14.321352799 +0000 UTC m=+39.899768764" watchObservedRunningTime="2026-10-31 02:55:14.321826934 +0000 UTC m=+39.900242896" Oct 31 02:55:14.399121 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.398282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:14.399121 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:14.398501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:14.399121 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:14.398607 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:14.399121 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:14.398664 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:16.398645993 +0000 UTC m=+41.977061953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : configmap "kube-rbac-proxy" not found Oct 31 02:55:14.399121 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:14.399037 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:14.399121 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:14.399081 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:16.399067518 +0000 UTC m=+41.977483475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:16.184898 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:16.184866 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:16.185265 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:16.184922 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:16.185302 ip-10-0-130-12 kubenswrapper[2576]: I1031 02:55:16.185284 2576 scope.go:122] "RemoveContainer" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" Oct 31 02:55:16.185490 ip-10-0-130-12 kubenswrapper[2576]: E1031 02:55:16.185469 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:16.242277 ip-10-0-130-12 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 02:55:16.251630 ip-10-0-130-12 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 02:55:16.251795 ip-10-0-130-12 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 02:55:16.251962 ip-10-0-130-12 systemd[1]: kubelet.service: Consumed 2.872s CPU time. Oct 31 02:55:16.262261 ip-10-0-130-12 systemd[1]: Starting Kubernetes Kubelet... Oct 31 02:55:16.364298 ip-10-0-130-12 kubenswrapper[4854]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:16.364298 ip-10-0-130-12 kubenswrapper[4854]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 02:55:16.364298 ip-10-0-130-12 kubenswrapper[4854]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:16.364298 ip-10-0-130-12 kubenswrapper[4854]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:16.366395 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366310 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.366395 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366395 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366400 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366402 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366405 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366409 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366411 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366414 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366418 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366423 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366426 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366428 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366431 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366434 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366436 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366439 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366442 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366444 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366447 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366450 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.366454 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366452 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366462 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366465 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366468 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366470 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366473 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366475 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366478 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366480 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366483 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366485 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366488 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366490 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366494 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366496 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366499 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366502 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366505 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366508 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366510 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.366912 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366513 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366515 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366518 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366521 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366523 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366526 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366530 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366532 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366535 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366538 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366540 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366543 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366546 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366549 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366551 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366554 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366556 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366559 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366561 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366564 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.367428 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366566 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366569 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366571 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366574 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366577 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366580 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366584 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366587 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366590 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366593 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366596 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366599 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366602 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366605 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366607 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366610 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366612 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366615 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366617 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.367924 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366620 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366623 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366625 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366628 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366630 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366634 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366636 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366640 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366642 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366645 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366647 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366650 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366652 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366655 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366658 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366660 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366663 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366667 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366670 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.368409 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366673 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366676 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366679 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366681 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366684 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366686 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366689 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.366692 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367472 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367480 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367484 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367487 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367490 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367493 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367495 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367498 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367503 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367508 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367512 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.368887 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367516 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367520 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367523 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367525 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367528 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367531 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367533 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367536 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367539 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367541 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367544 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367547 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367550 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367552 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367558 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367561 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367563 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367565 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367568 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.369350 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367571 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367573 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367576 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367578 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367581 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367584 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367586 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367589 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367591 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367594 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367597 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367599 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367602 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367605 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367607 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367610 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367613 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367616 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367618 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367621 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.369850 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367623 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367626 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367628 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367631 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367633 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367636 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367639 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367641 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367645 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367647 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367650 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367653 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367655 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367662 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367665 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367667 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367670 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367672 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367675 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367677 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.370359 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367680 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367683 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367685 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367688 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367691 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367693 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367696 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367699 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367701 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367704 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367707 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367710 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367712 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367715 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367717 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367720 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367722 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367725 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367727 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.370881 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367730 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367733 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367735 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367738 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367741 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367743 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367747 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367749 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367752 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367755 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367757 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367760 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367762 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367765 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367767 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367770 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.367773 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368260 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368266 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368270 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.371347 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368273 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368276 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368279 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368282 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368285 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368288 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368291 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368294 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368296 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368299 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368302 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368304 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368307 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368309 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368312 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368315 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368317 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368320 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368323 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368327 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.371815 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368330 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368333 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368335 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368338 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368341 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368343 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368346 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368349 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368351 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368353 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368356 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368359 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368362 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368364 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368367 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368370 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368372 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368375 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368377 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368380 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.372326 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368382 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368385 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368388 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368391 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368394 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368396 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368399 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368402 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368405 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368407 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368410 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368414 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368416 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368419 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368422 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368424 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368427 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368430 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368432 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368435 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.372851 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368437 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368440 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368442 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368445 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368449 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368453 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368456 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368459 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368462 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368464 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368467 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368470 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368473 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368476 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368478 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368481 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368484 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368487 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368489 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.373652 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368492 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368500 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368503 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368506 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368510 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368513 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368515 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368518 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368520 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368523 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368525 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368529 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368532 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368536 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368540 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368542 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368545 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368548 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368551 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.374170 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368553 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368556 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368559 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368561 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.368564 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368652 4854 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368664 4854 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368670 4854 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368674 4854 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368679 4854 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368682 4854 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368686 4854 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368690 4854 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368693 4854 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368696 4854 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368705 4854 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368708 4854 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368713 4854 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368716 4854 flags.go:64] FLAG: --cgroup-root="" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368719 4854 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368722 4854 flags.go:64] FLAG: --client-ca-file="" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368725 4854 flags.go:64] FLAG: --cloud-provider="external" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368728 4854 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 02:55:16.374661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368735 4854 flags.go:64] FLAG: --cluster-domain="" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368739 4854 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368742 4854 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368745 4854 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368749 4854 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368752 4854 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368755 4854 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368758 4854 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368762 4854 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368765 4854 flags.go:64] FLAG: --contention-profiling="false" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368768 4854 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368770 4854 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368774 4854 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368776 4854 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368781 4854 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368784 4854 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368787 4854 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368790 4854 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368792 4854 flags.go:64] FLAG: --enable-server="true" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368795 4854 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368802 4854 flags.go:64] FLAG: --event-burst="100" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368805 4854 flags.go:64] FLAG: --event-qps="50" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368808 4854 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368811 4854 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 02:55:16.375214 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368814 4854 flags.go:64] FLAG: --eviction-hard="" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368825 4854 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368830 4854 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368833 4854 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368836 4854 flags.go:64] FLAG: --eviction-soft="" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368839 4854 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368842 4854 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368845 4854 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368848 4854 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368852 4854 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368855 4854 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368858 4854 flags.go:64] FLAG: --feature-gates="" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368862 4854 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368865 4854 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368868 4854 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368871 4854 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368874 4854 flags.go:64] FLAG: --healthz-port="10248" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368877 4854 flags.go:64] FLAG: --help="false" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368879 4854 flags.go:64] FLAG: --hostname-override="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368883 4854 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368886 4854 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368888 4854 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368892 4854 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 02:55:16.375833 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368895 4854 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368898 4854 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368901 4854 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368904 4854 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368907 4854 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368909 4854 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368912 4854 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368915 4854 flags.go:64] FLAG: --kube-reserved="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368918 4854 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368921 4854 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368924 4854 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368928 4854 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368931 4854 flags.go:64] FLAG: --lock-file="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368934 4854 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368937 4854 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368940 4854 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368945 4854 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368948 4854 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368953 4854 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368956 4854 flags.go:64] FLAG: --logging-format="text" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368959 4854 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368962 4854 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368966 4854 flags.go:64] FLAG: --manifest-url="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368969 4854 flags.go:64] FLAG: --manifest-url-header="" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368974 4854 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 02:55:16.376408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368977 4854 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368980 4854 flags.go:64] FLAG: --max-pods="110" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368983 4854 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368986 4854 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368989 4854 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368992 4854 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368995 4854 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.368997 4854 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369000 4854 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369007 4854 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369010 4854 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369013 4854 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369016 4854 flags.go:64] FLAG: --pod-cidr="" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369019 4854 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369021 4854 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369024 4854 flags.go:64] FLAG: --pods-per-core="0" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369027 4854 flags.go:64] FLAG: --port="10250" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369030 4854 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369034 4854 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0f9a2308d368ae1e0" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369040 4854 flags.go:64] FLAG: --qos-reserved="" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369043 4854 flags.go:64] FLAG: --read-only-port="10255" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369046 4854 flags.go:64] FLAG: --register-node="true" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369049 4854 flags.go:64] FLAG: --register-with-taints="" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369053 4854 flags.go:64] FLAG: --registry-burst="10" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369056 4854 flags.go:64] FLAG: --registry-qps="5" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369060 4854 flags.go:64] FLAG: --reserved-cpus="" Oct 31 02:55:16.377025 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369063 4854 flags.go:64] FLAG: --reserved-memory="" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369067 4854 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369070 4854 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369073 4854 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369076 4854 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369079 4854 flags.go:64] FLAG: --runonce="false" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369082 4854 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369085 4854 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369088 4854 flags.go:64] FLAG: --seccomp-default="false" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369091 4854 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369093 4854 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369096 4854 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369099 4854 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369102 4854 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369105 4854 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369108 4854 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369110 4854 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369113 4854 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369116 4854 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369119 4854 flags.go:64] FLAG: --system-cgroups="" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369121 4854 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369126 4854 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369129 4854 flags.go:64] FLAG: --tls-cert-file="" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369132 4854 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369140 4854 flags.go:64] FLAG: --tls-min-version="" Oct 31 02:55:16.377674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369143 4854 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369151 4854 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369154 4854 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369157 4854 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369160 4854 flags.go:64] FLAG: --v="2" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369164 4854 flags.go:64] FLAG: --version="false" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369170 4854 flags.go:64] FLAG: --vmodule="" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369174 4854 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369177 4854 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369327 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369331 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369334 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369337 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369340 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369342 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369345 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369348 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369351 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369354 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369357 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369359 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369362 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.378273 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369365 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369367 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369370 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369372 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369375 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369377 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369380 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369383 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369385 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369390 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369392 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369395 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369398 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369401 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369403 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369406 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369409 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369412 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369414 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.378798 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369417 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369420 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369422 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369424 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369427 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369429 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369432 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369434 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369437 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369439 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369442 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369445 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369448 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369450 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369453 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369455 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369458 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369460 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369463 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369465 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.379259 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369468 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369471 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369474 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369477 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369480 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369482 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369486 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369489 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369491 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369496 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369498 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369501 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369503 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369506 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369509 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369511 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369514 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369516 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369519 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369521 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.379739 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369524 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369526 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369529 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369531 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369534 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369537 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369539 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369542 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369544 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369547 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369549 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369552 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369554 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369558 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369565 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369568 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369570 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369573 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369576 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.380287 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369579 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369583 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369588 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369591 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369594 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369597 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369599 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369602 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369604 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369607 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369609 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369612 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369615 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369617 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369620 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.380756 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.369646 4854 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369680 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369685 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369688 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369691 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369694 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369696 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369699 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369702 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369704 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369707 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369710 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369712 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369717 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369719 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369722 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369725 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369727 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369731 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369735 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.381153 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369746 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369749 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369752 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369754 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369757 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369760 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369762 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369765 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369767 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369770 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369773 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369775 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369778 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369781 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369783 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369786 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369789 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369792 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369794 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.381671 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369797 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369799 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369802 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369804 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369807 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369810 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369814 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369816 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369819 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369822 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369824 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369827 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369831 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369834 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369836 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369839 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369842 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369845 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369848 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369850 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.382167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369853 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369856 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369858 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369861 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369864 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369866 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369869 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369871 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369874 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369877 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369879 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369882 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369884 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369887 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369889 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369892 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369895 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369898 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369902 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.382874 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369905 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369908 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369910 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369913 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369915 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369920 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369923 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369927 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369930 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369933 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369936 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369938 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369941 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369943 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369945 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369948 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369950 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369953 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369955 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.383448 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369958 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369961 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369963 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369965 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369968 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369971 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369973 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369977 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369979 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.369981 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.375229 4854 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.375262 4854 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.375284 4854 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.375289 4854 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375348 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375354 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375357 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375360 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375364 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.383905 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375368 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375372 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375375 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375378 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375380 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375383 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375386 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375389 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375391 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375394 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375397 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375400 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375402 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375405 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375408 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375411 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375413 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375416 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375419 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375422 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.384501 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375425 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375427 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375430 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375433 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375436 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375438 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375441 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375444 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375446 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375449 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375452 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375454 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375457 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375460 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375462 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375465 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375467 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375470 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375472 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375476 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.384979 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375479 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375482 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375484 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375487 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375489 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375492 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375495 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375497 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375500 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375502 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375505 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375508 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375510 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375513 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375516 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375518 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375521 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375523 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375527 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375529 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.385465 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375532 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375534 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375537 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375539 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375542 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375544 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375547 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375550 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375552 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375555 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375557 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375560 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375563 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375566 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375576 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375579 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375582 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375584 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375587 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375589 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.385945 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375592 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375596 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375599 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375603 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375606 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375608 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375611 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375614 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375616 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375619 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375622 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375625 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375627 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375630 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375633 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375635 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375638 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375641 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375643 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.386483 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375646 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375649 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.375674 4854 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375805 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375809 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375814 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375818 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375821 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375825 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375828 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375831 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375834 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375837 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375839 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375842 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.386921 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375845 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375848 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375850 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375853 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375856 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375858 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375861 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375864 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375867 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375869 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375872 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375875 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375877 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375880 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375882 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375885 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375888 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375890 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375893 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375896 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.387291 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375899 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375901 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375904 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375906 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375909 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375911 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375914 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375917 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375919 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375922 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375925 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375927 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375930 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375932 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375935 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375938 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375940 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375943 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375945 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375948 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.387770 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375950 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375953 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375955 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375958 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375960 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375963 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375965 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375968 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375970 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375973 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375975 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375979 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375982 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375985 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375988 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375991 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375993 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375996 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.375998 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.388241 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376001 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376003 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376006 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376009 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376011 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376014 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376016 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376019 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376021 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376024 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376027 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376029 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376032 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376035 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376037 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376040 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376042 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376045 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376047 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376050 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.388737 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376052 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376055 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376057 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376060 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376062 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376065 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376067 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376070 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376072 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376075 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376078 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376080 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376083 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376086 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376088 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.376111 4854 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:16.389203 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376155 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376160 4854 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376163 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376166 4854 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376169 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376172 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376175 4854 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376178 4854 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376180 4854 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376183 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376186 4854 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376188 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376191 4854 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376194 4854 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376196 4854 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376199 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376201 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376204 4854 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376206 4854 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376209 4854 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:16.389662 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376211 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376215 4854 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376220 4854 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376224 4854 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376228 4854 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376231 4854 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376234 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376236 4854 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376239 4854 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376242 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376261 4854 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376264 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376269 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376272 4854 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376275 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376277 4854 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376280 4854 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376283 4854 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376286 4854 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376288 4854 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:16.390167 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376291 4854 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376295 4854 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376299 4854 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376301 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376304 4854 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376306 4854 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376309 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376312 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376314 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376317 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376319 4854 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376322 4854 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376325 4854 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376329 4854 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376332 4854 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376334 4854 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376337 4854 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376340 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376342 4854 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:16.390660 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376345 4854 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376347 4854 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376350 4854 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376353 4854 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376356 4854 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376360 4854 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376375 4854 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376378 4854 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376381 4854 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376384 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376387 4854 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376389 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376392 4854 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376395 4854 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376397 4854 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376400 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376402 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376405 4854 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376408 4854 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376411 4854 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:16.391098 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376414 4854 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376416 4854 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376419 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376422 4854 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376425 4854 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376427 4854 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376431 4854 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376433 4854 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376436 4854 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376438 4854 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376441 4854 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376443 4854 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376446 4854 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376449 4854 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376451 4854 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376454 4854 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376456 4854 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376460 4854 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376462 4854 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:16.391708 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376465 4854 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376468 4854 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376470 4854 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376473 4854 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376477 4854 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376480 4854 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376483 4854 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:16.376486 4854 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.376542 4854 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.377518 4854 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.377564 4854 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.377749 4854 server.go:1014] "Starting client certificate rotation" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.377909 4854 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.377947 4854 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 02:49:35 +0000 UTC" deadline="2028-04-05 22:58:10.202784838 +0000 UTC" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.377992 4854 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12548h2m53.824794561s" Oct 31 02:55:16.392162 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.378196 4854 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:55:16.392750 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.379243 4854 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:55:16.392750 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.380512 4854 log.go:25] "Validated CRI v1 runtime API" Oct 31 02:55:16.392750 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.382094 4854 log.go:25] "Validated CRI v1 image API" Oct 31 02:55:16.392750 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.382932 4854 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 02:55:16.392750 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.384984 4854 fs.go:135] Filesystem UUIDs: map[15d6fb13-90ef-421b-be10-1fea330abcf6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fcfed4ff-01a3-45ff-8067-e1f6f3845409:/dev/nvme0n1p3] Oct 31 02:55:16.392995 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.384999 4854 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a04461bfea6f53e187034091ed891d992dc1a3897e00b3453c980b12b92678b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a04461bfea6f53e187034091ed891d992dc1a3897e00b3453c980b12b92678b/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f536b103ce85f26096da3b73015998c0ef840587ce49fd3f77dec3aa6f8ab94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f536b103ce85f26096da3b73015998c0ef840587ce49fd3f77dec3aa6f8ab94/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/468fbf8a27df2d7b14e59052b97ae41f195509544f67e39e96a56f849fa05f28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/468fbf8a27df2d7b14e59052b97ae41f195509544f67e39e96a56f849fa05f28/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fbdd3ce7a3372e444b27cf74755411e927d85ddfb8d1e05be748a075d7cd5b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fbdd3ce7a3372e444b27cf74755411e927d85ddfb8d1e05be748a075d7cd5b2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05ce07aa-7ab8-45da-85c1-f036dc52c6b8/volumes/kubernetes.io~projected/kube-api-access-28qbk:{mountpoint:/var/lib/kubelet/pods/05ce07aa-7ab8-45da-85c1-f036dc52c6b8/volumes/kubernetes.io~projected/kube-api-access-28qbk major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/111372d5-9b74-4bae-b96f-3671787ad5b3/volumes/kubernetes.io~projected/kube-api-access-m5xb6:{mountpoint:/var/lib/kubelet/pods/111372d5-9b74-4bae-b96f-3671787ad5b3/volumes/kubernetes.io~projected/kube-api-access-m5xb6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~projected/kube-api-access-brs2h:{mountpoint:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~projected/kube-api-access-brs2h major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22caca16-dc54-4bac-8dff-b87cab1b8b60/volumes/kubernetes.io~projected/kube-api-access-g4rvq:{mountpoint:/var/lib/kubelet/pods/22caca16-dc54-4bac-8dff-b87cab1b8b60/volumes/kubernetes.io~projected/kube-api-access-g4rvq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/volumes/kubernetes.io~projected/kube-api-access-7r8ch:{mountpoint:/var/lib/kubelet/pods/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/volumes/kubernetes.io~projected/kube-api-access-7r8ch major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/volumes/kubernetes.io~secret/serving-cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e/volumes/kubernetes.io~projected/kube-api-access-5cq9z:{mountpoint:/var/lib/kubelet/pods/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e/volumes/kubernetes.io~projected/kube-api-access-5cq9z major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/566be63b-5b2b-4f4f-a40f-1e83e7837f74/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/566be63b-5b2b-4f4f-a40f-1e83e7837f74/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e948ca8-1809-4eeb-b014-f563cb6fdb91/volumes/kubernetes.io~projected/kube-api-access-6snh2:{mountpoint:/var/lib/kubelet/pods/5e948ca8-1809-4eeb-b014-f563cb6fdb91/volumes/kubernetes.io~projected/kube-api-access-6snh2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/616e84dc-1530-44cd-b4b3-752e5635a0ea/volumes/kubernetes.io~projected/kube-api-access-t9t8c:{mountpoint:/var/lib/kubelet/pods/616e84dc-1530-44cd-b4b3-752e5635a0ea/volumes/kubernetes.io~projected/kube-api-access-t9t8c major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64b281bc-7922-46b5-8d33-2f1bfdebb8ec/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/64b281bc-7922-46b5-8d33-2f1bfdebb8ec/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc/volumes/kubernetes.io~projected/kube-api-access-6x7mb:{mountpoint:/var/lib/kubelet/pods/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc/volumes/kubernetes.io~projected/kube-api-access-6x7mb major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5/volumes/kubernetes.io~projected/kube-api-access-pvcmb:{mountpoint:/var/lib/kubelet/pods/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5/volumes/kubernetes.io~projected/kube-api-access-pvcmb major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d72e18-bf2d-46e9-91ef-2374a0d41cf1/volumes/kubernetes.io~projected/kube-api-access-xxbmw:{mountpoint:/var/lib/kubelet/pods/70d72e18-bf2d-46e9-91ef-2374a0d41cf1/volumes/kubernetes.io~projected/kube-api-access-xxbmw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84351d2c-2829-4e33-afa0-0f3f813dd4fa/volumes/kubernetes.io~projected/kube-api-access-qqhf8:{mountpoint:/var/lib/kubelet/pods/84351d2c-2829-4e33-afa0-0f3f813dd4fa/volumes/kubernetes.io~projected/kube-api-access-qqhf8 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~projected/kube-api-access-jlxrw:{mountpoint:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~projected/kube-api-access-jlxrw major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94b41492-1a35-4ede-adcd-65e4c41c6cc0/volumes/kubernetes.io~projected/kube-api-access-mgj7m:{mountpoint:/var/lib/kubelet/pods/94b41492-1a35-4ede-adcd-65e4c41c6cc0/volumes/kubernetes.io~projected/kube-api-access-mgj7m major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9888db69-6183-403a-82cf-01ad3667c6f9/volumes/kubernetes.io~projected/kube-api-access-pphtw:{mountpoint:/var/lib/kubelet/pods/9888db69-6183-403a-82cf-01ad3667c6f9/volumes/kubernetes.io~projected/kube-api-access-pphtw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aca0a795-839c-4055-b009-793e1ffc4d58/volumes/kubernetes.io~projected/kube-api-access-lf2nn:{mountpoint:/var/lib/kubelet/pods/aca0a795-839c-4055-b009-793e1ffc4d58/volumes/kubernetes.io~projected/kube-api-access-lf2nn major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3be8a54-63a4-408c-ae91-e21314a80325/volumes/kubernetes.io~projected/kube-api-access-r75tc:{mountpoint:/var/lib/kubelet/pods/b3be8a54-63a4-408c-ae91-e21314a80325/volumes/kubernetes.io~projected/kube-api-access-r75tc major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b408b194-9df2-4518-b247-0d9a83d80f3b/volumes/kubernetes.io~projected/kube-api-access-5txxr:{mountpoint:/var/lib/kubelet/pods/b408b194-9df2-4518-b247-0d9a83d80f3b/volumes/kubernetes.io~projected/kube-api-access-5txxr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~projected/kube-api-access-95qbt:{mountpoint:/var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~projected/kube-api-access-95qbt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~projected/kube-api-access-p4cvz:{mountpoint:/var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~projected/kube-api-access-p4cvz major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~secret/default-certificate major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~secret/stats-auth major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4987385-e27b-4220-92d4-2f61bdb1ebc8/volumes/kubernetes.io~projected/kube-api-access-24c76:{mountpoint:/var/lib/kubelet/pods/c4987385-e27b-4220-92d4-2f61bdb1ebc8/volumes/kubernetes.io~projected/kube-api-access-24c76 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volumes/kubernetes.io~projected/kube-api-access-5n6p6:{mountpoint:/var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volumes/kubernetes.io~projected/kube-api-access-5n6p6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7194787-0fa9-43a4-a749-c6aac9433cfc/volumes/kubernetes.io~projected/kube-api-access-5z2k9:{mountpoint:/var/lib/kubelet/pods/e7194787-0fa9-43a4-a749-c6aac9433cfc/volumes/kubernetes.io~projected/kube-api-access-5z2k9 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7194787-0fa9-43a4-a749-c6aac9433cfc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e7194787-0fa9-43a4-a749-c6aac9433cfc/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8f27649-b353-4181-bc11-3aa1ec714be3/volumes/kubernetes.io~projected/kube-api-access-tp2tx:{mountpoint:/var/lib/kubelet/pods/e8f27649-b353-4181-bc11-3aa1ec714be3/volumes/kubernetes.io~projected/kube-api-access-tp2tx major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95878f0-c838-42c0-a546-55f075727d45/volumes/kubernetes.io~projected/kube-api-access-7qpnn:{mountpoint:/var/lib/kubelet/pods/e95878f0-c838-42c0-a546-55f075727d45/volumes/kubernetes.io~projected/kube-api-access-7qpnn major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95878f0-c838-42c0-a546-55f075727d45/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e95878f0-c838-42c0-a546-55f075727d45/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec216fff-d06e-4a0a-b35e-73b3a2d513a8/volumes/kubernetes.io~projected/kube-api-access-gsdbc:{mountpoint:/var/lib/kubelet/pods/ec216fff-d06e-4a0a-b35e-73b3a2d513a8/volumes/kubernetes.io~projected/kube-api-access-gsdbc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec216fff-d06e-4a0a-b35e-73b3a2d513a8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ec216fff-d06e-4a0a-b35e-73b3a2d513a8/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb123961-0503-4503-b9d4-6790d658e12e/volumes/kubernetes.io~projected/kube-api-access-vcs5b:{mountpoint:/var/lib/kubelet/pods/fb123961-0503-4503-b9d4-6790d658e12e/volumes/kubernetes.io~projected/kube-api-access-vcs5b major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb123961-0503-4503-b9d4-6790d658e12e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fb123961-0503-4503-b9d4-6790d658e12e/volumes/kubernetes.io~secret/cert major:0 minor:235 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/86d1a04920d1e5cc51d531e9fc7e73d4c176acb9f579469f6fad7f7fe6d50c70/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e85db5b0b7a2eb4baae553e7a6c69649af41f8c21037a3fc05e44a8616b70e3c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5e4ad04500165addcf1852432c71a29993892906a4112f412cfcfb10a60a620f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ed07a206f2614ba639c5efbc6ec49d7797625b4eaaf486d02f83079676e03900/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bced7abcdb0f87eb127f9ccc9e7dae24472ab4c239f0705562101e39dcbd8175/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9e1e1d4a4bacd272fd52cf4c66020bf48e69344bcf41867d36c02a76a6a097f1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/da234a990cee70c6921dff034906ce01fbd3d5307c8cf7603c80bfa66f2ebf68/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f1949e51d0f5a3ca8886d02e7acda996f5054dfd6206f09b55096bee3c9865f4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ded069a1c50da5e030bdeaff88972831d55619f5085eb709ae9f0e671588da90/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/80998e58ce81552e58eeb29263104113d57e9302553267502128646e0391ace9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6d1a528fa1da9974c1f0d08b6ef6d656de2e19c0152dd8a1108b822162d8219f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/55b4d205eb6adf0b9a9145c5b5947bcb9194868136bea3aaa7c6b537b6c73204/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/1e4c7027cb410330c4617d72dbee5bbdeb2f006edf169ab8755b6cda82575b02/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/6350c2f90235cc329c68c153d3be69626d7790f93a5dba9e80bef520ed07c9df/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/8f71f1af361a0393e8c1f2f471400095e40290756214ccd33320e59ce8b4416a/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/aca7f7d94126042ed4c4a53006c9c674239abd5718bf2ad3520a214567d5234d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2717289b653b7ed8d093b8385b2edd2e51e1aeb9bc6400d1ec75f883217115be/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4bf01788f120cb5d8250611b4c8c18a2134dbbb0c6026f86713c10202f017c83/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6d4b696a078f0ea3472679e859dccd1d6dfd38cfed4273dc5ce919733dc55335/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c2fc4de25ff7e492303814232bbeda2410ba13200e7b3ad12f51c008c3798275/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/36a8136bc28bc336451ba4cc43d906e02049528ddc3488b16788640c6c8848fd/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/d8adf7e6cac73295e57f32e6a46b5d3840580d8ce4301d503b347644c677398d/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ff4b4be9312d6c44a92c7737d63becf6c0b1174c65861eac0f1a790d9b465356/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/9e1bdcef7272dcf2f095b0657c2629233e9ad7cc00da7e96b0b2df648fee1ff8/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/b2311bccd0bb7d423f1f336cc68a314160e617c993cbd1bb280b133fe681d8ce/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3ac64921a564cdbda6634dab654439ef614440ce7d087c304ff36239f69dcbb2/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f28c96a5092ec6f81aab134cb41843631fa08659e55e099463622bd4eea4add8/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a1175ecfd1fee4f0aab221a82094f0953b54c861026dc5f1e28503cf65f71a23/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/fc37a9f90abc70286c397747de22e5a64b8513417214c373a55ed83515332db0/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/7fbcdf3d113c8d7131f980997e3a462e977fc4ad05482d27534f4ece5ed5af3a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/5e77b8476c46d7176f39826b1e6d7973299449ae35f3a73c8515b907fe55d158/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/1e8dce711706e49eca44942a547b45b64c45ffe3556f24532bc84e4b778c9b5a/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/45d7dea35cc6274678ef7aaf4853c33fe8ea299a2cf8569b19777db3bcd01613/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/e2c38a837e8339ac54dee101f0542c83c7fae7ba35528ce1a0d4395af885271d/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/a243ef2abedf858cbecf1d57ab8a23c3780099deaf6fe76c5422381f56fbdc78/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/2076c13c1516e5a8489e6a1f099613d7bd1404c2bfa43d94e23fd16958001f8d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/821f28e42557b4ccea38c1049a8c9dfa78f3dbeb38dc4cf3d7b8a4b9fa65bf5f/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/5a4da1f062ac321f6228b7e72015c1adc51f04993672b96f5c1c4ea7d6e66783/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/bab92fc8e02a915335c67138600a8314203f28edcc605960018472d7f6abbc77/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/67b740c69f585bc0e951c691aee55d80834d8d12ef706b2e6f3774c0503cb43d/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/29215ce053be1082a2efd81e45908910923b4fa7005b4f3781fca0e0bfc8ef88/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/18f5b8792ebf932f06df780029aa5e65e169e4780afb726aaa60d04ebbc63593/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/467eda725a41799489264be1971100828653fe7c398b6562b85a731b3d2f7d81/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/d2ee90ed86c24e0c4f71d29af1f0887bf78b4f594af73fe9be6cbd1ffef0e7fd/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/aa80e6ba1f1592b935ec0908fa0370eca7d88e12132f5ec8ba710961ea839065/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/887535cfb4e21a00b2259d76ec966b22003f90db578bc38f4c7a5e54c999c800/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/199299099d5da29465fc10e47ce6a43b5695afdd62be3bb9a7f54cb0e012cf09/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/986144641c7019aa6245b40935e3164e620d6cea3254260b3767dd1464aaed24/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/395e917ef389f54ba7d982fd3ad78da646faaa68820395ba6701d65a11123fb0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/23e003d76dd49734f85da8605dda7530411edbaabae05dc0eb19a328d302c6d8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c1988deeb4fea939075d0c1d2f7bbea7a986f31a847f944991572d3ea7d32f76/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/998c45ef4d21d8583524ef2d32f8352e98e929c409b569cbd6315cd5ef26b7fa/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5fe33492bb6a20ca420028f6f373c5e324dbdac6bc6e3ee7e59541f1a25970df/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/07c10d6a2c6acec1cf394f038b928337e7c1b08bc600844e841be2ae494df3cd/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 02:55:16.394864 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.394545 4854 manager.go:217] Machine: {Timestamp:2026-10-31 02:55:16.394058167 +0000 UTC m=+0.091225461 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099455 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d67836b784544d88235e5caddba0b SystemUUID:ec2d6783-6b78-4544-d882-35e5caddba0b BootID:2412d30f-e801-411b-8699-be74b247ab13 Filesystems:[{Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/566be63b-5b2b-4f4f-a40f-1e83e7837f74/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b408b194-9df2-4518-b247-0d9a83d80f3b/volumes/kubernetes.io~projected/kube-api-access-5txxr DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5e948ca8-1809-4eeb-b014-f563cb6fdb91/volumes/kubernetes.io~projected/kube-api-access-6snh2 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc/volumes/kubernetes.io~projected/kube-api-access-6x7mb DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e/volumes/kubernetes.io~projected/kube-api-access-5cq9z DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ec216fff-d06e-4a0a-b35e-73b3a2d513a8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fbdd3ce7a3372e444b27cf74755411e927d85ddfb8d1e05be748a075d7cd5b2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec6ef233d3cc53e0677ef7893d2ec7c0f6c2dc222bcb74509f86ce570a49d556/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7194787-0fa9-43a4-a749-c6aac9433cfc/volumes/kubernetes.io~projected/kube-api-access-5z2k9 DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/111372d5-9b74-4bae-b96f-3671787ad5b3/volumes/kubernetes.io~projected/kube-api-access-m5xb6 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9888db69-6183-403a-82cf-01ad3667c6f9/volumes/kubernetes.io~projected/kube-api-access-pphtw DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7194787-0fa9-43a4-a749-c6aac9433cfc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b57e3db9d85da252716c8f19586213f06a0b3e0e9cb58ff4327acf4332ffc9a/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb123961-0503-4503-b9d4-6790d658e12e/volumes/kubernetes.io~projected/kube-api-access-vcs5b DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/84351d2c-2829-4e33-afa0-0f3f813dd4fa/volumes/kubernetes.io~projected/kube-api-access-qqhf8 DeviceMajor:0 DeviceMinor:306 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/94b41492-1a35-4ede-adcd-65e4c41c6cc0/volumes/kubernetes.io~projected/kube-api-access-mgj7m DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3be8a54-63a4-408c-ae91-e21314a80325/volumes/kubernetes.io~projected/kube-api-access-r75tc DeviceMajor:0 DeviceMinor:238 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/70d72e18-bf2d-46e9-91ef-2374a0d41cf1/volumes/kubernetes.io~projected/kube-api-access-xxbmw DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7f10f85e4fbd71c5f7a8704d17c77290d2fa0a12c26533faa5359d9fba6b061/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6a69de233b263e7b257023c3238704a5e1747cf8337b65c3cc4a62133a69ae3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3d2b198967e9c0d2645d9bba46d4c67e7d1759395a538052e0758242296a7b7/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e95878f0-c838-42c0-a546-55f075727d45/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81a05074fa8839152fbced8d0932e80ced0cac28f1966c1913cd1dc31da34d7e/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6bcfabaa00eab530f48ba3b08476b1f60330a7303875916f3b721b89ca4313d/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~projected/kube-api-access-p4cvz DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8b152a09b2e4fece35cbe37baa5a321862fa2f2c56d4467b14b54d621f2fb24/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05ce07aa-7ab8-45da-85c1-f036dc52c6b8/volumes/kubernetes.io~projected/kube-api-access-28qbk DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec216fff-d06e-4a0a-b35e-73b3a2d513a8/volumes/kubernetes.io~projected/kube-api-access-gsdbc DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~projected/kube-api-access-jlxrw DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aca0a795-839c-4055-b009-793e1ffc4d58/volumes/kubernetes.io~projected/kube-api-access-lf2nn DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4987385-e27b-4220-92d4-2f61bdb1ebc8/volumes/kubernetes.io~projected/kube-api-access-24c76 DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f536b103ce85f26096da3b73015998c0ef840587ce49fd3f77dec3aa6f8ab94/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ec54c7695c6a8d02f8f1279a26172149e46f1d2d83ccfa69d0b53d853b3b850/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e36c114784ca2e2775e491d177f9d8a7674ddeedb769e30cd6c91c793f201afe/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86191885aaaa7999d054205079f0d7711aa5fbf9209c381a039c615597a80c6d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~projected/kube-api-access-brs2h DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:234 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dcf9a703-8b57-41ae-a801-cee51331a479/volumes/kubernetes.io~projected/kube-api-access-5n6p6 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/22caca16-dc54-4bac-8dff-b87cab1b8b60/volumes/kubernetes.io~projected/kube-api-access-g4rvq DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2468e1fd62d46f7744e1e85d72efe8a35a43781136dc8ed9a48ac7348d94c03/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8f27649-b353-4181-bc11-3aa1ec714be3/volumes/kubernetes.io~projected/kube-api-access-tp2tx DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64b281bc-7922-46b5-8d33-2f1bfdebb8ec/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb123961-0503-4503-b9d4-6790d658e12e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:235 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ec0e5fc6fbfedbb2535199f528f5909eb062a531fdfed6e32699d5c3bbbb280/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/468fbf8a27df2d7b14e59052b97ae41f195509544f67e39e96a56f849fa05f28/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/616e84dc-1530-44cd-b4b3-752e5635a0ea/volumes/kubernetes.io~projected/kube-api-access-t9t8c DeviceMajor:0 DeviceMinor:237 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5/volumes/kubernetes.io~projected/kube-api-access-pvcmb DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e95878f0-c838-42c0-a546-55f075727d45/volumes/kubernetes.io~projected/kube-api-access-7qpnn DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/518bafbf93416b02ff6403c1641076c4e5745732e4955cb9ff79c714d8d2abcf/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a04461bfea6f53e187034091ed891d992dc1a3897e00b3453c980b12b92678b/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24a6bfc9b4bdc5a3e0d0fe90049ed90c786a0b55c1663de60fc1332856f46111/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b467f911679af3c5b0fc1ff68f45f188de966603764ee771ddfb2765f0e03e1/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bfe2e7-b724-42de-acff-a45a1609c7d0/volumes/kubernetes.io~projected/kube-api-access-95qbt DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/volumes/kubernetes.io~projected/kube-api-access-7r8ch DeviceMajor:0 DeviceMinor:239 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ec54c7695c6a8d MacAddress:5e:90:07:24:33:5e Speed:10000 Mtu:8901} {Name:1a04461bfea6f53 MacAddress:2a:6a:ec:dd:f8:8f Speed:10000 Mtu:8901} {Name:1f536b103ce85f2 MacAddress:de:df:cf:22:1e:05 Speed:10000 Mtu:8901} {Name:468fbf8a27df2d7 MacAddress:a6:78:79:f5:0e:9f Speed:10000 Mtu:8901} {Name:518bafbf93416b0 MacAddress:a6:af:6e:b1:9c:ce Speed:10000 Mtu:8901} {Name:81a05074fa88391 MacAddress:a2:12:dc:51:eb:5c Speed:10000 Mtu:8901} {Name:9b467f911679af3 MacAddress:be:86:2f:6e:e4:3b Speed:10000 Mtu:8901} {Name:9b57e3db9d85da2 MacAddress:ee:9a:ca:24:31:b0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f8:0f:4f:b5 Speed:0 Mtu:9001} {Name:br-int MacAddress:26:30:cc:fa:61:66 Speed:0 Mtu:8901} {Name:c8b152a09b2e4fe MacAddress:f2:27:8b:2a:69:d3 Speed:10000 Mtu:8901} {Name:e36c114784ca2e2 MacAddress:32:9e:a4:99:50:da Speed:10000 Mtu:8901} {Name:e6bcfabaa00eab5 MacAddress:22:cf:8e:19:45:62 Speed:10000 Mtu:8901} {Name:ec6ef233d3cc53e MacAddress:8e:6d:3f:74:8b:d2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f8:0f:4f:b5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7e:7a:06:ff:8b:5e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:ac:28:f1:6d:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 02:55:16.394864 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.394868 4854 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 02:55:16.395132 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.394908 4854 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 02:55:16.395191 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395169 4854 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 02:55:16.395356 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395192 4854 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-12.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 02:55:16.395398 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395359 4854 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 02:55:16.395398 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395367 4854 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 02:55:16.395398 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395376 4854 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:55:16.395398 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395389 4854 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:55:16.395500 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395474 4854 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 02:55:16.395639 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395630 4854 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 02:55:16.395744 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395735 4854 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 02:55:16.395776 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395769 4854 kubelet.go:493] "Attempting to sync node with API server" Oct 31 02:55:16.395806 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395786 4854 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 02:55:16.395806 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395796 4854 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 02:55:16.395806 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395804 4854 kubelet.go:405] "Adding apiserver pod source" Oct 31 02:55:16.395897 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.395812 4854 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 02:55:16.396514 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.396499 4854 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 02:55:16.396514 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.396511 4854 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:55:16.397014 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.396997 4854 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 02:55:16.399447 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399432 4854 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 02:55:16.399447 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399444 4854 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:55:16.399541 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399534 4854 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 02:55:16.399719 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399708 4854 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 02:55:16.399749 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399737 4854 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 02:55:16.399824 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399815 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399829 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399835 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399840 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399846 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399851 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399858 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 02:55:16.399861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399863 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 02:55:16.400041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399871 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 02:55:16.400041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399877 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 02:55:16.400041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399890 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 02:55:16.400041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399898 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 02:55:16.400041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399920 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 02:55:16.400041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.399927 4854 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 02:55:16.403471 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.403456 4854 server.go:1263] "Started kubelet" Oct 31 02:55:16.403614 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.403563 4854 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 02:55:16.403675 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.403593 4854 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 02:55:16.403675 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.403647 4854 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 02:55:16.404045 ip-10-0-130-12 systemd[1]: Started Kubernetes Kubelet. Oct 31 02:55:16.404857 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.404319 4854 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 02:55:16.405658 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.405641 4854 server.go:317] "Adding debug handlers to kubelet server" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.408067 4854 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.410096 4854 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.410336 4854 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:37 +0000 UTC" deadline="2028-04-06 07:01:02.546523323 +0000 UTC" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.410371 4854 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12556h5m46.136155381s" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.410802 4854 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.410816 4854 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 02:55:16.411181 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.410976 4854 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 02:55:16.414405 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414380 4854 factory.go:55] Registering systemd factory Oct 31 02:55:16.414405 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414403 4854 factory.go:223] Registration of the systemd container factory successfully Oct 31 02:55:16.414607 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:16.414592 4854 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-12.us-east-2.compute.internal\" not found" Oct 31 02:55:16.414684 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414672 4854 factory.go:153] Registering CRI-O factory Oct 31 02:55:16.414732 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414690 4854 factory.go:223] Registration of the crio container factory successfully Oct 31 02:55:16.414783 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414746 4854 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 02:55:16.414783 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414767 4854 factory.go:103] Registering Raw factory Oct 31 02:55:16.414783 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.414783 4854 manager.go:1199] Started watching for new ooms in manager Oct 31 02:55:16.415199 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.415186 4854 manager.go:319] Starting recovery of all containers Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416189 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416219 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9888db69-6183-403a-82cf-01ad3667c6f9" volumeName="kubernetes.io/projected/9888db69-6183-403a-82cf-01ad3667c6f9-kube-api-access-pphtw" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416231 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8" volumeName="kubernetes.io/projected/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-kube-api-access-7r8ch" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416263 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" volumeName="kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416275 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa634f6-8f9c-4cd6-946d-bc018c3c20bc" volumeName="kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416308 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416321 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416334 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416348 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9888db69-6183-403a-82cf-01ad3667c6f9" volumeName="kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-cni-binary-copy" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416359 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" volumeName="kubernetes.io/projected/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-kube-api-access-gsdbc" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416370 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416382 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3be8a54-63a4-408c-ae91-e21314a80325" volumeName="kubernetes.io/projected/b3be8a54-63a4-408c-ae91-e21314a80325-kube-api-access-r75tc" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416393 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7194787-0fa9-43a4-a749-c6aac9433cfc" volumeName="kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-tmp" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416404 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7194787-0fa9-43a4-a749-c6aac9433cfc" volumeName="kubernetes.io/projected/e7194787-0fa9-43a4-a749-c6aac9433cfc-kube-api-access-5z2k9" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416425 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95878f0-c838-42c0-a546-55f075727d45" volumeName="kubernetes.io/secret/e95878f0-c838-42c0-a546-55f075727d45-serving-cert" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416449 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb123961-0503-4503-b9d4-6790d658e12e" volumeName="kubernetes.io/secret/fb123961-0503-4503-b9d4-6790d658e12e-cert" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416463 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ce07aa-7ab8-45da-85c1-f036dc52c6b8" volumeName="kubernetes.io/projected/05ce07aa-7ab8-45da-85c1-f036dc52c6b8-kube-api-access-28qbk" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416475 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416486 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcf9a703-8b57-41ae-a801-cee51331a479" volumeName="kubernetes.io/projected/dcf9a703-8b57-41ae-a801-cee51331a479-kube-api-access-5n6p6" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416521 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bfe2e7-b724-42de-acff-a45a1609c7d0" volumeName="kubernetes.io/projected/b6bfe2e7-b724-42de-acff-a45a1609c7d0-kube-api-access-95qbt" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416544 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcf9a703-8b57-41ae-a801-cee51331a479" volumeName="kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-script-lib" seLinuxMountContext="" Oct 31 02:55:16.416539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416554 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcf9a703-8b57-41ae-a801-cee51331a479" volumeName="kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-env-overrides" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416565 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" volumeName="kubernetes.io/secret/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-serving-cert" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416577 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bfe2e7-b724-42de-acff-a45a1609c7d0" volumeName="kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-tmp" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416588 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d72e18-bf2d-46e9-91ef-2374a0d41cf1" volumeName="kubernetes.io/configmap/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-iptables-alerter-script" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416597 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416607 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64b281bc-7922-46b5-8d33-2f1bfdebb8ec" volumeName="kubernetes.io/secret/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-agent-certs" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416619 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca0a795-839c-4055-b009-793e1ffc4d58" volumeName="kubernetes.io/projected/aca0a795-839c-4055-b009-793e1ffc4d58-kube-api-access-lf2nn" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416649 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7194787-0fa9-43a4-a749-c6aac9433cfc" volumeName="kubernetes.io/secret/e7194787-0fa9-43a4-a749-c6aac9433cfc-serving-cert" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416660 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5" volumeName="kubernetes.io/configmap/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-telemetry-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416671 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416681 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64b281bc-7922-46b5-8d33-2f1bfdebb8ec" volumeName="kubernetes.io/configmap/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-konnectivity-ca" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416692 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b408b194-9df2-4518-b247-0d9a83d80f3b" volumeName="kubernetes.io/configmap/b408b194-9df2-4518-b247-0d9a83d80f3b-serviceca" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416702 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfdd5bc7-bedb-4abc-96ec-9e7d9f803702" volumeName="kubernetes.io/projected/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-kube-api-access-p4cvz" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416737 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8f27649-b353-4181-bc11-3aa1ec714be3" volumeName="kubernetes.io/projected/e8f27649-b353-4181-bc11-3aa1ec714be3-kube-api-access-tp2tx" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416753 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95878f0-c838-42c0-a546-55f075727d45" volumeName="kubernetes.io/configmap/e95878f0-c838-42c0-a546-55f075727d45-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416766 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" volumeName="kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416801 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416812 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b408b194-9df2-4518-b247-0d9a83d80f3b" volumeName="kubernetes.io/projected/b408b194-9df2-4518-b247-0d9a83d80f3b-kube-api-access-5txxr" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416824 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfdd5bc7-bedb-4abc-96ec-9e7d9f803702" volumeName="kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-default-certificate" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416838 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcf9a703-8b57-41ae-a801-cee51331a479" volumeName="kubernetes.io/secret/dcf9a703-8b57-41ae-a801-cee51331a479-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416851 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb123961-0503-4503-b9d4-6790d658e12e" volumeName="kubernetes.io/projected/fb123961-0503-4503-b9d4-6790d658e12e-kube-api-access-vcs5b" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416862 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5" volumeName="kubernetes.io/projected/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-kube-api-access-pvcmb" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416872 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416883 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b41492-1a35-4ede-adcd-65e4c41c6cc0" volumeName="kubernetes.io/projected/94b41492-1a35-4ede-adcd-65e4c41c6cc0-kube-api-access-mgj7m" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416895 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3be8a54-63a4-408c-ae91-e21314a80325" volumeName="kubernetes.io/configmap/b3be8a54-63a4-408c-ae91-e21314a80325-config-volume" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416906 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7194787-0fa9-43a4-a749-c6aac9433cfc" volumeName="kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-service-ca-bundle" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416920 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416933 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22caca16-dc54-4bac-8dff-b87cab1b8b60" volumeName="kubernetes.io/projected/22caca16-dc54-4bac-8dff-b87cab1b8b60-kube-api-access-g4rvq" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416945 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3be8a54-63a4-408c-ae91-e21314a80325" volumeName="kubernetes.io/empty-dir/b3be8a54-63a4-408c-ae91-e21314a80325-tmp-dir" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416958 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" volumeName="kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-trusted-ca" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416971 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8" volumeName="kubernetes.io/secret/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-serving-cert" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416982 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca0a795-839c-4055-b009-793e1ffc4d58" volumeName="kubernetes.io/configmap/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.416995 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bfe2e7-b724-42de-acff-a45a1609c7d0" volumeName="kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-tuned" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417007 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcf9a703-8b57-41ae-a801-cee51331a479" volumeName="kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417021 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7194787-0fa9-43a4-a749-c6aac9433cfc" volumeName="kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-snapshots" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417041 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" volumeName="kubernetes.io/projected/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-kube-api-access-5cq9z" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417054 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48eda75e-e883-4929-8278-fd49da0f5b27" volumeName="kubernetes.io/configmap/48eda75e-e883-4929-8278-fd49da0f5b27-nginx-conf" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417066 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84351d2c-2829-4e33-afa0-0f3f813dd4fa" volumeName="kubernetes.io/empty-dir/84351d2c-2829-4e33-afa0-0f3f813dd4fa-data-volume" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417080 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eb2947c-46ed-4f8d-a837-d1004e3ac459" volumeName="kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417094 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfdd5bc7-bedb-4abc-96ec-9e7d9f803702" volumeName="kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-stats-auth" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417105 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417117 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8" volumeName="kubernetes.io/configmap/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417129 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e948ca8-1809-4eeb-b014-f563cb6fdb91" volumeName="kubernetes.io/empty-dir/5e948ca8-1809-4eeb-b014-f563cb6fdb91-tmp-dir" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417142 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e948ca8-1809-4eeb-b014-f563cb6fdb91" volumeName="kubernetes.io/projected/5e948ca8-1809-4eeb-b014-f563cb6fdb91-kube-api-access-6snh2" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417155 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9888db69-6183-403a-82cf-01ad3667c6f9" volumeName="kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-multus-daemon-config" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417166 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4987385-e27b-4220-92d4-2f61bdb1ebc8" volumeName="kubernetes.io/projected/c4987385-e27b-4220-92d4-2f61bdb1ebc8-kube-api-access-24c76" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417178 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111372d5-9b74-4bae-b96f-3671787ad5b3" volumeName="kubernetes.io/projected/111372d5-9b74-4bae-b96f-3671787ad5b3-kube-api-access-m5xb6" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417188 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1181c942-8297-48bf-b860-16d8dd8f6a8a" volumeName="kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417200 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84351d2c-2829-4e33-afa0-0f3f813dd4fa" volumeName="kubernetes.io/projected/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-api-access-qqhf8" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417212 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" volumeName="kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-binary-copy" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417225 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="566be63b-5b2b-4f4f-a40f-1e83e7837f74" volumeName="kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417236 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="616e84dc-1530-44cd-b4b3-752e5635a0ea" volumeName="kubernetes.io/projected/616e84dc-1530-44cd-b4b3-752e5635a0ea-kube-api-access-t9t8c" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417268 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7194787-0fa9-43a4-a749-c6aac9433cfc" volumeName="kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-trusted-ca-bundle" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417280 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95878f0-c838-42c0-a546-55f075727d45" volumeName="kubernetes.io/projected/e95878f0-c838-42c0-a546-55f075727d45-kube-api-access-7qpnn" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417291 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" volumeName="kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417302 4854 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d72e18-bf2d-46e9-91ef-2374a0d41cf1" volumeName="kubernetes.io/projected/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-kube-api-access-xxbmw" seLinuxMountContext="" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417315 4854 reconstruct.go:97] "Volume reconstruction finished" Oct 31 02:55:16.417424 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.417321 4854 reconciler.go:29] "Reconciler: start to sync state" Oct 31 02:55:16.419997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.419201 4854 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:16.419997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.419519 4854 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:16.419997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.419920 4854 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:16.420324 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.420308 4854 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 02:55:16.428038 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.428011 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/1.log" Oct 31 02:55:16.428344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.428328 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/1.log" Oct 31 02:55:16.447110 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.447083 4854 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 02:55:16.448513 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.448492 4854 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 02:55:16.448600 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.448521 4854 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 02:55:16.448600 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.448543 4854 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 02:55:16.448600 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:16.448581 4854 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 02:55:16.472126 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.472097 4854 generic.go:356] "Generic (PLEG): container finished" podID="ea9031209d866497c6611ade6539778f" containerID="af4d86591d77c4052810d072d3b8224fcca83f37fa6cab3ecd751a84552f8238" exitCode=0 Oct 31 02:55:16.474921 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.474900 4854 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:16.491832 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.491808 4854 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="5579075ff4753da6cf4c746d52b668bd2a851089131833e544f3120785a73973" exitCode=0 Oct 31 02:55:16.491832 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.491829 4854 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="153134343ca5eaea75d9099509bbd77de62e51aa3cf9447035934f543d7aeeb0" exitCode=0 Oct 31 02:55:16.491832 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.491835 4854 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="0db4c2583cbbdde8b388f3f9919e90bd64d7a5047747a5a202d19d4e12e89800" exitCode=0 Oct 31 02:55:16.492041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.491841 4854 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="c770db6c5aeb97accd028f6bb0aa4d0c01cd6c90b8b38a839585a3f75b3497fd" exitCode=0 Oct 31 02:55:16.492041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.491846 4854 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="023e23fad68c64b14e777f3d277976031f16510eb852983b7ed5d0097d8e885e" exitCode=0 Oct 31 02:55:16.492041 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.491851 4854 generic.go:356] "Generic (PLEG): container finished" podID="3cd31a28-7ec3-4d09-ac91-e060a5e2d56e" containerID="412085e90f40ed06da097d6f351f96c51838bc531ffed9447245c5dc9f0a4f9d" exitCode=0 Oct 31 02:55:16.498627 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.498575 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/1.log" Oct 31 02:55:16.498627 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.498610 4854 generic.go:356] "Generic (PLEG): container finished" podID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" exitCode=255 Oct 31 02:55:16.505412 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.505399 4854 manager.go:324] Recovery completed Oct 31 02:55:16.524827 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524806 4854 cpu_manager.go:226] "Starting" policy="none" Oct 31 02:55:16.524827 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524820 4854 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 02:55:16.524949 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524836 4854 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 02:55:16.524990 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524947 4854 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 02:55:16.524990 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524963 4854 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 02:55:16.524990 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524979 4854 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 02:55:16.524990 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524984 4854 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 02:55:16.524990 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524989 4854 policy_none.go:50] "Start" Oct 31 02:55:16.525117 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.524997 4854 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 02:55:16.525117 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.525006 4854 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 02:55:16.525170 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.525144 4854 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 02:55:16.525170 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.525152 4854 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 02:55:16.525170 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.525157 4854 policy_none.go:44] "Start" Oct 31 02:55:16.529387 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.529361 4854 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 02:55:16.532346 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.532332 4854 manager.go:341] "Starting Device Plugin manager" Oct 31 02:55:16.532857 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:16.532358 4854 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 02:55:16.532857 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.532857 4854 server.go:87] "Starting device plugin registration server" Oct 31 02:55:16.533105 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533093 4854 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 02:55:16.533169 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533103 4854 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 02:55:16.533264 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533205 4854 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 02:55:16.533436 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533393 4854 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 02:55:16.533495 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533446 4854 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 02:55:16.533495 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533453 4854 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 02:55:16.533604 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.533495 4854 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T02:55:16.533433674Z","UUID":"63fac679-cb9e-4736-8c09-fffd88b56015","Handler":null,"Name":"","Endpoint":""} Oct 31 02:55:16.534502 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:16.534482 4854 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 02:55:16.534811 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.534794 4854 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 02:55:16.534888 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.534823 4854 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 02:55:16.548681 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.548648 4854 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal"] Oct 31 02:55:16.549200 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549173 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"ea9031209d866497c6611ade6539778f","Type":"ContainerStarted","Data":"e629a732b0670531ba396298993c4a1e8deed62d54a3e32eaef44c24bbaf4b1a"} Oct 31 02:55:16.549263 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549207 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"ea9031209d866497c6611ade6539778f","Type":"ContainerDied","Data":"af4d86591d77c4052810d072d3b8224fcca83f37fa6cab3ecd751a84552f8238"} Oct 31 02:55:16.549263 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549222 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"ea9031209d866497c6611ade6539778f","Type":"ContainerStarted","Data":"f10417ee2f382859239826696ecac202c31c58f9edba3be0ded9ceaf676311fc"} Oct 31 02:55:16.549332 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549291 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"915a6fb768d2fd4c56c8c1c0d6da80b2","Type":"ContainerStarted","Data":"1a35190fb070366149190d749e50fb30895d2ad2e158c08243c0eb7b0bdabc9a"} Oct 31 02:55:16.549332 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549304 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" event={"ID":"915a6fb768d2fd4c56c8c1c0d6da80b2","Type":"ContainerStarted","Data":"7e0fdf0005595868a09ff755749b94ec4093c761896fbcdb6bd2e4be04a743e6"} Oct 31 02:55:16.549332 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549321 4854 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.549441 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.549321 4854 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.564697 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.564679 4854 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:16.564809 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.564713 4854 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:16.564809 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:16.564724 4854 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.564809 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:16.564756 4854 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.621061 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.621035 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.621161 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.621070 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.621161 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.621097 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/915a6fb768d2fd4c56c8c1c0d6da80b2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"915a6fb768d2fd4c56c8c1c0d6da80b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.634159 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.634137 4854 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:55:16.635322 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.635300 4854 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:55:16.635411 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.635329 4854 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:55:16.635411 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.635340 4854 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-12.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:55:16.635411 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.635385 4854 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.646486 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.646471 4854 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.646557 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.646521 4854 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.721350 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.721324 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.721430 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.721388 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.721430 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.721406 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.721430 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.721422 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea9031209d866497c6611ade6539778f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"ea9031209d866497c6611ade6539778f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.721573 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.721458 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/915a6fb768d2fd4c56c8c1c0d6da80b2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"915a6fb768d2fd4c56c8c1c0d6da80b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:16.721573 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:16.721505 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/915a6fb768d2fd4c56c8c1c0d6da80b2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" (UID: \"915a6fb768d2fd4c56c8c1c0d6da80b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:17.396464 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.396432 4854 apiserver.go:52] "Watching apiserver" Oct 31 02:55:17.404566 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.404533 4854 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 02:55:17.406606 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.406573 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sjjkf","openshift-ingress/router-default-66695d6bd7-5pk65","openshift-ovn-kubernetes/ovnkube-node-5nsnk","openshift-cluster-node-tuning-operator/tuned-7vws7","openshift-insights/insights-operator-7c4986d449-xth74","openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj","openshift-image-registry/image-registry-75b45ffb98-wkq9k","kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-j5zbk","openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-rsvkk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf","openshift-network-operator/iptables-alerter-jpxqb","kube-system/global-pull-secret-syncer-4pp9z","openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw","openshift-multus/multus-9rzxm","openshift-image-registry/node-ca-8nv7d","openshift-ingress-canary/ingress-canary-ss5vz","openshift-network-console/networking-console-plugin-86c478bb88-5k4sc","openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46","kube-system/kas-connection-checker-685895c7c9-9n9mr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn","openshift-dns/dns-default-pwpgn","openshift-multus/multus-additional-cni-plugins-qrb4n","openshift-multus/network-metrics-daemon-5qlrx","kube-system/kas-connection-checker-685895c7c9-8mfvv","openshift-console-operator/console-operator-7cd56b8dbc-l8j7j","openshift-network-diagnostics/network-check-target-r76hd","openshift-dns/node-resolver-2ltlq"] Oct 31 02:55:17.407886 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.407867 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.408728 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.408711 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.409348 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.409322 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.410101 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.410084 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.411226 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.411205 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:17.411619 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.411602 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:17.417862 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.417841 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 02:55:17.417977 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.417927 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.418131 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418116 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkjm6\"" Oct 31 02:55:17.418211 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418201 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hv8gl\"" Oct 31 02:55:17.418622 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418600 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.418736 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418623 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.418736 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418608 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.418736 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418606 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.418736 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.418688 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.419174 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.419152 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mzmdp\"" Oct 31 02:55:17.419296 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.419230 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.419422 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.419406 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 02:55:17.419567 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.419545 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 02:55:17.419691 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.419673 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.420179 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.420159 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 02:55:17.421194 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.421169 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 02:55:17.421382 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.421364 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 02:55:17.421634 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.421620 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 02:55:17.421881 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.421660 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 02:55:17.421968 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.421934 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jp47q\"" Oct 31 02:55:17.422180 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.422159 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 02:55:17.422312 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.422297 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.422975 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.422952 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 02:55:17.423065 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423017 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.423353 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423331 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.423485 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423469 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wr2d\"" Oct 31 02:55:17.423947 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.422362 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 02:55:17.424056 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423524 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 02:55:17.424113 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423022 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.424113 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423502 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.424113 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.423345 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.424745 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424728 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 02:55:17.424745 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424738 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 02:55:17.424850 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424755 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 02:55:17.424850 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424823 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 02:55:17.424850 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424829 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 02:55:17.424850 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424841 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424866 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424879 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424888 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424909 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424909 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-mqqj2\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424924 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424925 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gxlr7\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424944 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-49z5f\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424953 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 02:55:17.424978 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424955 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424985 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x5rg4\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.424998 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sts47\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425012 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425024 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425034 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425062 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425069 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425077 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2b67f\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425102 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425114 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425116 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425132 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425150 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425177 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425187 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls2x2\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425205 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425225 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425238 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425262 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425287 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425295 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425305 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425341 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.425344 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425340 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425365 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jsfq8\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425376 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-tmp\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425385 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425420 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425468 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sc997\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425472 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-tmp\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425476 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-serving-cert\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425504 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425522 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425633 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9t8c\" (UniqueName: \"kubernetes.io/projected/616e84dc-1530-44cd-b4b3-752e5635a0ea-kube-api-access-t9t8c\") pod \"network-check-source-6f78d59f99-zp7jw\" (UID: \"616e84dc-1530-44cd-b4b3-752e5635a0ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425654 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-serving-cert\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425668 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425667 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425678 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425697 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c76\" (UniqueName: \"kubernetes.io/projected/c4987385-e27b-4220-92d4-2f61bdb1ebc8-kube-api-access-24c76\") pod \"kas-connection-checker-685895c7c9-j5zbk\" (UID: \"c4987385-e27b-4220-92d4-2f61bdb1ebc8\") " pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425704 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425805 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425839 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425843 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2nn\" (UniqueName: \"kubernetes.io/projected/aca0a795-839c-4055-b009-793e1ffc4d58-kube-api-access-lf2nn\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425872 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-stats-auth\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425898 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpnn\" (UniqueName: \"kubernetes.io/projected/e95878f0-c838-42c0-a546-55f075727d45-kube-api-access-7qpnn\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425917 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-snapshots\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425933 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbpkr\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.425937 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7194787-0fa9-43a4-a749-c6aac9433cfc-serving-cert\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426056 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426077 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e7194787-0fa9-43a4-a749-c6aac9433cfc-serving-cert\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426081 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426103 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426127 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-stats-auth\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426135 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e7194787-0fa9-43a4-a749-c6aac9433cfc-snapshots\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426141 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qbk\" (UniqueName: \"kubernetes.io/projected/05ce07aa-7ab8-45da-85c1-f036dc52c6b8-kube-api-access-28qbk\") pod \"volume-data-source-validator-58c9f5f46d-xgpqk\" (UID: \"05ce07aa-7ab8-45da-85c1-f036dc52c6b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426164 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426173 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426213 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q5ns6\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426233 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426235 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7qc7w\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426282 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426288 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426328 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426341 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426373 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgj7m\" (UniqueName: \"kubernetes.io/projected/94b41492-1a35-4ede-adcd-65e4c41c6cc0-kube-api-access-mgj7m\") pod \"kas-connection-checker-685895c7c9-9n9mr\" (UID: \"94b41492-1a35-4ede-adcd-65e4c41c6cc0\") " pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426391 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e95878f0-c838-42c0-a546-55f075727d45-config\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:17.426392 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426419 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426438 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426445 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426467 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426502 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426516 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426589 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e95878f0-c838-42c0-a546-55f075727d45-config\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426601 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426636 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426668 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-default-certificate\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426717 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cvz\" (UniqueName: \"kubernetes.io/projected/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-kube-api-access-p4cvz\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426720 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426751 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8ch\" (UniqueName: \"kubernetes.io/projected/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-kube-api-access-7r8ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426786 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426803 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426813 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426817 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bfddz\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426837 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426870 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-config\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426870 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426907 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-default-certificate\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426916 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426926 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426953 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvcmb\" (UniqueName: \"kubernetes.io/projected/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-kube-api-access-pvcmb\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.426986 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2tx\" (UniqueName: \"kubernetes.io/projected/e8f27649-b353-4181-bc11-3aa1ec714be3-kube-api-access-tp2tx\") pod \"kas-connection-checker-685895c7c9-8mfvv\" (UID: \"e8f27649-b353-4181-bc11-3aa1ec714be3\") " pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427007 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427012 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95878f0-c838-42c0-a546-55f075727d45-serving-cert\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427010 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427011 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-config\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427039 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427088 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-service-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427114 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2k9\" (UniqueName: \"kubernetes.io/projected/e7194787-0fa9-43a4-a749-c6aac9433cfc-kube-api-access-5z2k9\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427139 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e95878f0-c838-42c0-a546-55f075727d45-serving-cert\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427158 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdbc\" (UniqueName: \"kubernetes.io/projected/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-kube-api-access-gsdbc\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427189 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wcv7f\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427289 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eda75e-e883-4929-8278-fd49da0f5b27-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427317 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-service-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427319 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvhtj\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427446 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48eda75e-e883-4929-8278-fd49da0f5b27-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.427642 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.428228 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428110 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428331 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428426 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428464 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428476 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428542 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428593 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428617 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428648 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x8dcs\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428703 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sgkvt\"" Oct 31 02:55:17.429373 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.428717 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 02:55:17.436849 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.436834 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.436897 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.436857 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.436939 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.436909 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.438693 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.438668 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.438753 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.438732 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7194787-0fa9-43a4-a749-c6aac9433cfc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.440654 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.440626 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9t8c\" (UniqueName: \"kubernetes.io/projected/616e84dc-1530-44cd-b4b3-752e5635a0ea-kube-api-access-t9t8c\") pod \"network-check-source-6f78d59f99-zp7jw\" (UID: \"616e84dc-1530-44cd-b4b3-752e5635a0ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zp7jw" Oct 31 02:55:17.440709 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.440690 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpnn\" (UniqueName: \"kubernetes.io/projected/e95878f0-c838-42c0-a546-55f075727d45-kube-api-access-7qpnn\") pod \"service-ca-operator-544d749c8c-5vq46\" (UID: \"e95878f0-c838-42c0-a546-55f075727d45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" Oct 31 02:55:17.440810 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.440796 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.440873 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.440858 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2nn\" (UniqueName: \"kubernetes.io/projected/aca0a795-839c-4055-b009-793e1ffc4d58-kube-api-access-lf2nn\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.441398 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.441383 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgj7m\" (UniqueName: \"kubernetes.io/projected/94b41492-1a35-4ede-adcd-65e4c41c6cc0-kube-api-access-mgj7m\") pod \"kas-connection-checker-685895c7c9-9n9mr\" (UID: \"94b41492-1a35-4ede-adcd-65e4c41c6cc0\") " pod="kube-system/kas-connection-checker-685895c7c9-9n9mr" Oct 31 02:55:17.447711 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.447685 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8ch\" (UniqueName: \"kubernetes.io/projected/32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8-kube-api-access-7r8ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf\" (UID: \"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" Oct 31 02:55:17.447822 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.447806 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2k9\" (UniqueName: \"kubernetes.io/projected/e7194787-0fa9-43a4-a749-c6aac9433cfc-kube-api-access-5z2k9\") pod \"insights-operator-7c4986d449-xth74\" (UID: \"e7194787-0fa9-43a4-a749-c6aac9433cfc\") " pod="openshift-insights/insights-operator-7c4986d449-xth74" Oct 31 02:55:17.447915 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.447900 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qbk\" (UniqueName: \"kubernetes.io/projected/05ce07aa-7ab8-45da-85c1-f036dc52c6b8-kube-api-access-28qbk\") pod \"volume-data-source-validator-58c9f5f46d-xgpqk\" (UID: \"05ce07aa-7ab8-45da-85c1-f036dc52c6b8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgpqk" Oct 31 02:55:17.448115 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.448097 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.448164 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.448092 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdbc\" (UniqueName: \"kubernetes.io/projected/ec216fff-d06e-4a0a-b35e-73b3a2d513a8-kube-api-access-gsdbc\") pod \"console-operator-7cd56b8dbc-l8j7j\" (UID: \"ec216fff-d06e-4a0a-b35e-73b3a2d513a8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:17.448701 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.448681 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvcmb\" (UniqueName: \"kubernetes.io/projected/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-kube-api-access-pvcmb\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.452640 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.452624 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cvz\" (UniqueName: \"kubernetes.io/projected/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-kube-api-access-p4cvz\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.452878 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.452859 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2tx\" (UniqueName: \"kubernetes.io/projected/e8f27649-b353-4181-bc11-3aa1ec714be3-kube-api-access-tp2tx\") pod \"kas-connection-checker-685895c7c9-8mfvv\" (UID: \"e8f27649-b353-4181-bc11-3aa1ec714be3\") " pod="kube-system/kas-connection-checker-685895c7c9-8mfvv" Oct 31 02:55:17.452913 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.452866 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c76\" (UniqueName: \"kubernetes.io/projected/c4987385-e27b-4220-92d4-2f61bdb1ebc8-kube-api-access-24c76\") pod \"kas-connection-checker-685895c7c9-j5zbk\" (UID: \"c4987385-e27b-4220-92d4-2f61bdb1ebc8\") " pod="kube-system/kas-connection-checker-685895c7c9-j5zbk" Oct 31 02:55:17.502456 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.502439 4854 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:17.502634 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.502621 4854 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:17.512073 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.512059 4854 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:17.512149 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.512097 4854 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:17.512546 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.512531 4854 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 02:55:17.524569 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.524549 4854 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:17.524637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.524584 4854 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-12.us-east-2.compute.internal" Oct 31 02:55:17.527843 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.527824 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:17.527935 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.527858 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.527935 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.527884 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xb6\" (UniqueName: \"kubernetes.io/projected/111372d5-9b74-4bae-b96f-3671787ad5b3-kube-api-access-m5xb6\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:17.527935 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.527909 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-var-lib-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.528022 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.527969 4854 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:17.528063 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528038 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-etc-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.528099 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.528084 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.028067389 +0000 UTC m=+1.725234692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:17.528135 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528101 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75tc\" (UniqueName: \"kubernetes.io/projected/b3be8a54-63a4-408c-ae91-e21314a80325-kube-api-access-r75tc\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.528135 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528125 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-netns\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.528197 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528142 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphtw\" (UniqueName: \"kubernetes.io/projected/9888db69-6183-403a-82cf-01ad3667c6f9-kube-api-access-pphtw\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.528197 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528184 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.528300 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528204 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-registration-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.528300 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528231 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysconfig\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.528382 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528299 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.528382 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528341 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:17.528473 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528424 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.528473 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528445 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.528569 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.528510 4854 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:17.528569 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528530 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.528569 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.528553 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.028542242 +0000 UTC m=+1.725709525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:17.528661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528601 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.528661 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528626 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.528729 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528673 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-kubelet\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.528729 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528697 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.528782 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528745 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-conf-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.528818 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528775 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.528870 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528813 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-kubelet\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.528919 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528886 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-multus-certs\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.528919 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528907 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/566be63b-5b2b-4f4f-a40f-1e83e7837f74-original-pull-secret\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.529008 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528918 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.529008 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528918 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.529008 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528961 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.529008 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528981 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.529008 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.528997 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529015 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529064 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529093 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529106 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-multus-daemon-config\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529199 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qbt\" (UniqueName: \"kubernetes.io/projected/b6bfe2e7-b724-42de-acff-a45a1609c7d0-kube-api-access-95qbt\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529232 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-host-slash\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529288 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cnibin\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.529314 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529292 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529335 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529379 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-var-lib-kubelet\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529400 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-multus-daemon-config\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529408 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.529425 4854 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529449 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.529475 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.029462899 +0000 UTC m=+1.726630211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529493 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-kubernetes\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529513 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6p6\" (UniqueName: \"kubernetes.io/projected/dcf9a703-8b57-41ae-a801-cee51331a479-kube-api-access-5n6p6\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529528 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529541 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-tmp\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529555 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84351d2c-2829-4e33-afa0-0f3f813dd4fa-crio-socket\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529569 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84351d2c-2829-4e33-afa0-0f3f813dd4fa-data-volume\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529585 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhf8\" (UniqueName: \"kubernetes.io/projected/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-api-access-qqhf8\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529601 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-netns\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529653 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-tmp\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529673 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529680 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84351d2c-2829-4e33-afa0-0f3f813dd4fa-data-volume\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529698 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-cni-binary-copy\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.529717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529721 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snh2\" (UniqueName: \"kubernetes.io/projected/5e948ca8-1809-4eeb-b014-f563cb6fdb91-kube-api-access-6snh2\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529754 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.529782 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.029769827 +0000 UTC m=+1.726937143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529800 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-konnectivity-ca\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529827 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-node-log\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529851 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-config\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529876 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3be8a54-63a4-408c-ae91-e21314a80325-config-volume\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529917 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9888db69-6183-403a-82cf-01ad3667c6f9-cni-binary-copy\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529917 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.529932 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.530004 4854 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530008 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530051 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530083 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-konnectivity-ca\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530086 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3be8a54-63a4-408c-ae91-e21314a80325-config-volume\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.530092 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.0300686 +0000 UTC m=+1.727235902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530114 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-config\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530117 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-bin\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530145 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-system-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530169 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-bin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530200 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b408b194-9df2-4518-b247-0d9a83d80f3b-host\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530228 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-host\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530265 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530311 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs5b\" (UniqueName: \"kubernetes.io/projected/fb123961-0503-4503-b9d4-6790d658e12e-kube-api-access-vcs5b\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530381 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-etc-kubernetes\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530426 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-multus\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.530438 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530450 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-systemd\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530489 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-iptables-alerter-script\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530553 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-dbus\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530663 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txxr\" (UniqueName: \"kubernetes.io/projected/b408b194-9df2-4518-b247-0d9a83d80f3b-kube-api-access-5txxr\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530696 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530736 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-k8s-cni-cncf-io\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530770 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-iptables-alerter-script\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.530797 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.530811 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530831 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-slash\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530860 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8a54-63a4-408c-ae91-e21314a80325-tmp-dir\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.530876 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.030866536 +0000 UTC m=+1.728033818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530917 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8a54-63a4-408c-ae91-e21314a80325-tmp-dir\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530947 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-modprobe-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530972 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-socket-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.530993 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-script-lib\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531071 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-sys\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531111 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531136 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-systemd-units\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531138 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-ovnkube-script-lib\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531200 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531238 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-device-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531283 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-hostroot\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531360 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e948ca8-1809-4eeb-b014-f563cb6fdb91-hosts-file\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531366 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531386 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531413 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf9a703-8b57-41ae-a801-cee51331a479-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.531470 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531432 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531547 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e948ca8-1809-4eeb-b014-f563cb6fdb91-tmp-dir\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531579 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf9a703-8b57-41ae-a801-cee51331a479-ovn-node-metrics-cert\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531602 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-agent-certs\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531650 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-systemd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531661 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e948ca8-1809-4eeb-b014-f563cb6fdb91-tmp-dir\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531720 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-log-socket\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531752 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-env-overrides\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531803 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-tuned\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531856 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531881 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-ovn\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531885 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dcf9a703-8b57-41ae-a801-cee51331a479-env-overrides\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531911 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64b281bc-7922-46b5-8d33-2f1bfdebb8ec-agent-certs\") pod \"konnectivity-agent-sjjkf\" (UID: \"64b281bc-7922-46b5-8d33-2f1bfdebb8ec\") " pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531930 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-etc-selinux\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531937 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-tuned\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531968 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-kubelet-config\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.531998 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b408b194-9df2-4518-b247-0d9a83d80f3b-serviceca\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532026 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxbmw\" (UniqueName: \"kubernetes.io/projected/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-kube-api-access-xxbmw\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532125 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rvq\" (UniqueName: \"kubernetes.io/projected/22caca16-dc54-4bac-8dff-b87cab1b8b60-kube-api-access-g4rvq\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532179 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b408b194-9df2-4518-b247-0d9a83d80f3b-serviceca\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532220 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-cnibin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532260 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cq9z\" (UniqueName: \"kubernetes.io/projected/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-kube-api-access-5cq9z\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.532343 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532291 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-socket-dir-parent\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532375 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-run\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532411 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-lib-modules\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532445 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-sys-fs\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532466 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb123961-0503-4503-b9d4-6790d658e12e-cert\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532528 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-conf\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532570 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-netd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.532970 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.532629 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb123961-0503-4503-b9d4-6790d658e12e-cert\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:17.537440 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.537421 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xb6\" (UniqueName: \"kubernetes.io/projected/111372d5-9b74-4bae-b96f-3671787ad5b3-kube-api-access-m5xb6\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:17.541336 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.541321 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75tc\" (UniqueName: \"kubernetes.io/projected/b3be8a54-63a4-408c-ae91-e21314a80325-kube-api-access-r75tc\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.541446 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.541432 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphtw\" (UniqueName: \"kubernetes.io/projected/9888db69-6183-403a-82cf-01ad3667c6f9-kube-api-access-pphtw\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.542141 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.542126 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.542656 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.542636 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.546231 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.546212 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxbmw\" (UniqueName: \"kubernetes.io/projected/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-kube-api-access-xxbmw\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.546231 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.546224 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txxr\" (UniqueName: \"kubernetes.io/projected/b408b194-9df2-4518-b247-0d9a83d80f3b-kube-api-access-5txxr\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.546563 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.546545 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rvq\" (UniqueName: \"kubernetes.io/projected/22caca16-dc54-4bac-8dff-b87cab1b8b60-kube-api-access-g4rvq\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.546615 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.546602 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cq9z\" (UniqueName: \"kubernetes.io/projected/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-kube-api-access-5cq9z\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.547756 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.547739 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snh2\" (UniqueName: \"kubernetes.io/projected/5e948ca8-1809-4eeb-b014-f563cb6fdb91-kube-api-access-6snh2\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.547998 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.547981 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7mb\" (UniqueName: \"kubernetes.io/projected/6aa634f6-8f9c-4cd6-946d-bc018c3c20bc-kube-api-access-6x7mb\") pod \"network-check-target-r76hd\" (UID: \"6aa634f6-8f9c-4cd6-946d-bc018c3c20bc\") " pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:17.548359 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.548341 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhf8\" (UniqueName: \"kubernetes.io/projected/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-api-access-qqhf8\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.548584 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.548569 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs5b\" (UniqueName: \"kubernetes.io/projected/fb123961-0503-4503-b9d4-6790d658e12e-kube-api-access-vcs5b\") pod \"ingress-canary-ss5vz\" (UID: \"fb123961-0503-4503-b9d4-6790d658e12e\") " pod="openshift-ingress-canary/ingress-canary-ss5vz" Oct 31 02:55:17.549105 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.549087 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qbt\" (UniqueName: \"kubernetes.io/projected/b6bfe2e7-b724-42de-acff-a45a1609c7d0-kube-api-access-95qbt\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.549697 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.549677 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6p6\" (UniqueName: \"kubernetes.io/projected/dcf9a703-8b57-41ae-a801-cee51331a479-kube-api-access-5n6p6\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.632872 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632853 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-kubelet\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.632977 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632877 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-multus-certs\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.632977 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632895 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.632977 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632913 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-host-slash\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.632977 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632949 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70d72e18-bf2d-46e9-91ef-2374a0d41cf1-host-slash\") pod \"iptables-alerter-jpxqb\" (UID: \"70d72e18-bf2d-46e9-91ef-2374a0d41cf1\") " pod="openshift-network-operator/iptables-alerter-jpxqb" Oct 31 02:55:17.632977 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632958 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-multus-certs\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632966 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-kubelet\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.632984 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cnibin\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633026 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-cnibin\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633032 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-var-lib-kubelet\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633052 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633073 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-var-lib-kubelet\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633078 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633100 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-kubernetes\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633117 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84351d2c-2829-4e33-afa0-0f3f813dd4fa-crio-socket\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633133 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-netns\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633141 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-kubernetes\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633112 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633154 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633170 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-netns\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633184 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-node-log\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633196 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633192 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84351d2c-2829-4e33-afa0-0f3f813dd4fa-crio-socket\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633219 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-node-log\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633230 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-bin\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633276 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-bin\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633276 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-system-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633320 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-system-cni-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633333 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-bin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633350 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b408b194-9df2-4518-b247-0d9a83d80f3b-host\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633363 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-host\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633391 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633410 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-bin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633418 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-etc-kubernetes\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633440 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84351d2c-2829-4e33-afa0-0f3f813dd4fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633447 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b408b194-9df2-4518-b247-0d9a83d80f3b-host\") pod \"node-ca-8nv7d\" (UID: \"b408b194-9df2-4518-b247-0d9a83d80f3b\") " pod="openshift-image-registry/node-ca-8nv7d" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633452 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633445 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-multus\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633472 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-host\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633474 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-etc-kubernetes\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633520 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-var-lib-cni-multus\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633552 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-systemd\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633571 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-dbus\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633595 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-k8s-cni-cncf-io\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633619 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-slash\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633626 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-systemd\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633642 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-modprobe-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633659 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-slash\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633658 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-dbus\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633667 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-socket-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633644 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-k8s-cni-cncf-io\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633693 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-sys\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633718 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633720 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-socket-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633719 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-modprobe-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633733 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-systemd-units\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633761 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-sys\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633772 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-systemd-units\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633778 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-device-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633802 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-hostroot\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.633808 4854 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633826 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e948ca8-1809-4eeb-b014-f563cb6fdb91-hosts-file\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633851 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.633865 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.133850098 +0000 UTC m=+1.831017397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633891 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-device-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633862 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-hostroot\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633893 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633894 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e948ca8-1809-4eeb-b014-f563cb6fdb91-hosts-file\") pod \"node-resolver-2ltlq\" (UID: \"5e948ca8-1809-4eeb-b014-f563cb6fdb91\") " pod="openshift-dns/node-resolver-2ltlq" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.633908 4854 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633952 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-systemd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.633932 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-systemd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.633941 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.633992 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:17.633997 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.634006 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.133979713 +0000 UTC m=+1.831146998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : secret "metrics-daemon-secret" not found Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.634041 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.134026408 +0000 UTC m=+1.831193691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634058 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-log-socket\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634078 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634094 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-ovn\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634110 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-etc-selinux\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634133 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-kubelet-config\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634152 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-log-socket\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634151 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cd31a28-7ec3-4d09-ac91-e060a5e2d56e-system-cni-dir\") pod \"multus-additional-cni-plugins-qrb4n\" (UID: \"3cd31a28-7ec3-4d09-ac91-e060a5e2d56e\") " pod="openshift-multus/multus-additional-cni-plugins-qrb4n" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634145 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-ovn\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634163 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-cnibin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634182 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-etc-selinux\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634200 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-socket-dir-parent\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634205 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-cnibin\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634204 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/566be63b-5b2b-4f4f-a40f-1e83e7837f74-kubelet-config\") pod \"global-pull-secret-syncer-4pp9z\" (UID: \"566be63b-5b2b-4f4f-a40f-1e83e7837f74\") " pod="kube-system/global-pull-secret-syncer-4pp9z" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634224 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-run\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634228 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-socket-dir-parent\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634263 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-lib-modules\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634275 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-run\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634292 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-sys-fs\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634312 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-conf\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634327 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-netd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634354 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-sys-fs\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634359 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634380 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-var-lib-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634395 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-etc-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634412 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-netns\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634436 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634438 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-cni-netd\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634444 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-conf\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634447 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysctl-d\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634461 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-registration-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634466 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-etc-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634473 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-var-lib-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634478 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-host-run-netns\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634436 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-lib-modules\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634486 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysconfig\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634510 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-registration-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634523 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6bfe2e7-b724-42de-acff-a45a1609c7d0-etc-sysconfig\") pod \"tuned-7vws7\" (UID: \"b6bfe2e7-b724-42de-acff-a45a1609c7d0\") " pod="openshift-cluster-node-tuning-operator/tuned-7vws7" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634487 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22caca16-dc54-4bac-8dff-b87cab1b8b60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hbrwn\" (UID: \"22caca16-dc54-4bac-8dff-b87cab1b8b60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hbrwn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634552 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634569 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634582 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-run-ovn-kubernetes\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634597 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-kubelet\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634614 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634628 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-conf-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.634635 4854 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:17.634662 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:18.134654018 +0000 UTC m=+1.831821299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634669 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9888db69-6183-403a-82cf-01ad3667c6f9-multus-conf-dir\") pod \"multus-9rzxm\" (UID: \"9888db69-6183-403a-82cf-01ad3667c6f9\") " pod="openshift-multus/multus-9rzxm" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634692 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-host-kubelet\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.635957 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.634693 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dcf9a703-8b57-41ae-a801-cee51331a479-run-openvswitch\") pod \"ovnkube-node-5nsnk\" (UID: \"dcf9a703-8b57-41ae-a801-cee51331a479\") " pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:17.709890 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:17.709836 4854 scope.go:122] "RemoveContainer" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" Oct 31 02:55:18.039237 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.039206 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:18.039404 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.039276 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:18.039404 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.039298 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:18.039404 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.039320 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:18.039404 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.039344 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:18.039404 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039376 4854 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039422 4854 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039431 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039445 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039448 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.039426859 +0000 UTC m=+2.736594141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039426 4854 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039470 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.039462338 +0000 UTC m=+2.736629620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:18.039588 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.039547 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:18.039800 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039596 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.039571321 +0000 UTC m=+2.736738611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:18.039800 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039617 4854 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:18.039800 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039619 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.039609194 +0000 UTC m=+2.736776476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:18.039800 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039635 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.039627439 +0000 UTC m=+2.736794726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:18.039800 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.039652 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.039641818 +0000 UTC m=+2.736809119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:18.139886 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.139861 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:18.139995 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.139961 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:18.139995 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.139983 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:18.139995 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.139987 4854 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.140000 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140040 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.140026819 +0000 UTC m=+2.837194111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140094 4854 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140098 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140114 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140120 4854 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140148 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.140133724 +0000 UTC m=+2.837301010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140169 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.140154394 +0000 UTC m=+2.837321683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : secret "metrics-daemon-secret" not found Oct 31 02:55:18.140219 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.140191 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.140181216 +0000 UTC m=+2.837348502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:18.507576 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.507554 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/2.log" Oct 31 02:55:18.507967 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.507936 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/1.log" Oct 31 02:55:18.508008 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.507966 4854 generic.go:356] "Generic (PLEG): container finished" podID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" containerID="e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38" exitCode=255 Oct 31 02:55:18.508068 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.508040 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" event={"ID":"ec216fff-d06e-4a0a-b35e-73b3a2d513a8","Type":"ContainerDied","Data":"e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38"} Oct 31 02:55:18.508121 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.508107 4854 scope.go:122] "RemoveContainer" containerID="8644cc477be1058f0756b04a2b72f664db08a5f3b0985fa1aad01cefef85bed7" Oct 31 02:55:18.508602 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:18.508573 4854 scope.go:122] "RemoveContainer" containerID="e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38" Oct 31 02:55:18.508779 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:18.508760 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:19.050407 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.050367 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.050417 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.050450 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.050477 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050510 4854 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050587 4854 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050614 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050591 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls podName:6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.050570637 +0000 UTC m=+4.747737942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fr9t2" (UID: "6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5") : secret "cluster-monitoring-operator-tls" not found Oct 31 02:55:19.050643 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050629 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050651 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.050634341 +0000 UTC m=+4.747801626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050672 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.050662446 +0000 UTC m=+4.747829765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.050695 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050724 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.050711572 +0000 UTC m=+4.747878859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050753 4854 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050798 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.050787862 +0000 UTC m=+4.747955147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050803 4854 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.050759 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:19.050904 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.050828 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.050821998 +0000 UTC m=+4.747989372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:19.151378 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.151355 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:19.151499 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.151384 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:19.151499 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.151402 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:19.151499 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.151459 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:19.151499 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151488 4854 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151531 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.151521041 +0000 UTC m=+4.848688322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : secret "metrics-daemon-secret" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151488 4854 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151566 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151581 4854 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151585 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151605 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.151592569 +0000 UTC m=+4.848759876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151620 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.151611443 +0000 UTC m=+4.848778726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:19.151637 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:19.151634 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:21.151627507 +0000 UTC m=+4.848794788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:19.513017 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:19.512989 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/2.log" Oct 31 02:55:21.031085 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.031032 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:21.069783 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.069755 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:21.069944 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.069803 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:21.069944 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.069895 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:21.069944 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.069909 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b45ffb98-wkq9k: secret "image-registry-tls" not found Oct 31 02:55:21.069944 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.069915 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.069954 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls podName:1181c942-8297-48bf-b860-16d8dd8f6a8a nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.069941747 +0000 UTC m=+8.767109032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls") pod "image-registry-75b45ffb98-wkq9k" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a") : secret "image-registry-tls" not found Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.069975 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.069996 4854 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.070026 4854 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.070038 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.070045 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.070033554 +0000 UTC m=+8.767200849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.070071 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.070058666 +0000 UTC m=+8.767225967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : secret "router-metrics-certs-default" not found Oct 31 02:55:21.070105 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.070105 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:21.070461 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.070118 4854 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 02:55:21.070461 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.070179 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls podName:aca0a795-839c-4055-b009-793e1ffc4d58 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.0701601 +0000 UTC m=+8.767327397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8qqdp" (UID: "aca0a795-839c-4055-b009-793e1ffc4d58") : secret "samples-operator-tls" not found Oct 31 02:55:21.070461 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.070208 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.070198443 +0000 UTC m=+8.767365774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:21.070461 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.070409 4854 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 02:55:21.072816 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.072798 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fr9t2\" (UID: \"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:21.171484 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.171455 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:21.171616 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.171534 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:21.171616 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.171555 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:21.171616 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.171572 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:21.171616 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171609 4854 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171650 4854 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171663 4854 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171669 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.171652536 +0000 UTC m=+8.868819817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171716 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs podName:111372d5-9b74-4bae-b96f-3671787ad5b3 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.171704841 +0000 UTC m=+8.868872137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs") pod "network-metrics-daemon-5qlrx" (UID: "111372d5-9b74-4bae-b96f-3671787ad5b3") : secret "metrics-daemon-secret" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171732 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.171722427 +0000 UTC m=+8.868889715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171745 4854 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 02:55:21.171775 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171761 4854 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj: secret "image-registry-tls" not found Oct 31 02:55:21.172009 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.171815 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls podName:8eb2947c-46ed-4f8d-a837-d1004e3ac459 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:25.171802849 +0000 UTC m=+8.868970130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls") pod "image-registry-7d8c6f6dc9-zfrlj" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459") : secret "image-registry-tls" not found Oct 31 02:55:21.309819 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.309748 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" Oct 31 02:55:21.579365 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.579294 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2"] Oct 31 02:55:21.583201 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:21.583171 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c03b70f_c1cc_4eb6_8cb6_10ab3a3cd1d5.slice/crio-1aaf58574c30ca08c12527054b757cd57353629daa2bd7185ca7a385d1d39682 WatchSource:0}: Error finding container 1aaf58574c30ca08c12527054b757cd57353629daa2bd7185ca7a385d1d39682: Status 404 returned error can't find the container with id 1aaf58574c30ca08c12527054b757cd57353629daa2bd7185ca7a385d1d39682 Oct 31 02:55:21.584964 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.584951 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:55:21.599941 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.599924 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:21.600213 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:21.600201 4854 scope.go:122] "RemoveContainer" containerID="e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38" Oct 31 02:55:21.600388 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:21.600371 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:22.141631 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:22.141599 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ltlq_5e948ca8-1809-4eeb-b014-f563cb6fdb91/dns-node-resolver/0.log" Oct 31 02:55:22.523520 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:22.523482 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" event={"ID":"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5","Type":"ContainerStarted","Data":"1aaf58574c30ca08c12527054b757cd57353629daa2bd7185ca7a385d1d39682"} Oct 31 02:55:23.141400 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:23.141372 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8nv7d_b408b194-9df2-4518-b247-0d9a83d80f3b/node-ca/0.log" Oct 31 02:55:23.528385 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:23.528351 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fr9t2" event={"ID":"6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5","Type":"ContainerStarted","Data":"df0f4356834825a79d19588e82f94053badcc15ee4ccc3e437faeeedbe2072c0"} Oct 31 02:55:23.690648 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:23.690618 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sjjkf" Oct 31 02:55:23.852616 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:23.852546 4854 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:23.852907 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:23.852895 4854 scope.go:122] "RemoveContainer" containerID="e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38" Oct 31 02:55:23.853081 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:23.853063 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:24.149789 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:24.149698 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ss5vz_fb123961-0503-4503-b9d4-6790d658e12e/serve-healthcheck-canary/0.log" Oct 31 02:55:24.943695 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:24.943666 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf_32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8/kube-storage-version-migrator-operator/0.log" Oct 31 02:55:25.113376 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.113334 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:25.113376 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.113392 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:25.113578 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.113508 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle podName:bfdd5bc7-bedb-4abc-96ec-9e7d9f803702 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:33.113491835 +0000 UTC m=+16.810659116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle") pod "router-default-66695d6bd7-5pk65" (UID: "bfdd5bc7-bedb-4abc-96ec-9e7d9f803702") : configmap references non-existent config key: service-ca.crt Oct 31 02:55:25.113578 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.113535 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:25.113673 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.113620 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:25.113673 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.113651 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:25.113805 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.113787 4854 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 02:55:25.114493 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.114240 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert podName:48eda75e-e883-4929-8278-fd49da0f5b27 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:33.114214271 +0000 UTC m=+16.811381625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5k4sc" (UID: "48eda75e-e883-4929-8278-fd49da0f5b27") : secret "networking-console-plugin-cert" not found Oct 31 02:55:25.117963 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.117924 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-metrics-certs\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:25.119791 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.119769 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aca0a795-839c-4055-b009-793e1ffc4d58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8qqdp\" (UID: \"aca0a795-839c-4055-b009-793e1ffc4d58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:25.120395 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.120378 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"image-registry-75b45ffb98-wkq9k\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:25.210515 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.210456 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:25.210618 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.210530 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" Oct 31 02:55:25.214524 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.214506 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:25.214589 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.214536 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:25.214589 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.214562 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:25.214690 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.214627 4854 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:25.214690 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.214673 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls podName:84351d2c-2829-4e33-afa0-0f3f813dd4fa nodeName:}" failed. No retries permitted until 2026-10-31 02:55:33.214659214 +0000 UTC m=+16.911826525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rsvkk" (UID: "84351d2c-2829-4e33-afa0-0f3f813dd4fa") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:25.214798 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.214749 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:25.214936 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.214919 4854 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:25.215010 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:25.214998 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls podName:b3be8a54-63a4-408c-ae91-e21314a80325 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:33.214981538 +0000 UTC m=+16.912148838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls") pod "dns-default-pwpgn" (UID: "b3be8a54-63a4-408c-ae91-e21314a80325") : secret "dns-default-metrics-tls" not found Oct 31 02:55:25.216923 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.216904 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/111372d5-9b74-4bae-b96f-3671787ad5b3-metrics-certs\") pod \"network-metrics-daemon-5qlrx\" (UID: \"111372d5-9b74-4bae-b96f-3671787ad5b3\") " pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:25.217447 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.217426 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"image-registry-7d8c6f6dc9-zfrlj\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:25.219210 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.219187 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:25.418345 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.418321 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:25.493199 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.493175 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp"] Oct 31 02:55:25.497315 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.497291 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75b45ffb98-wkq9k"] Oct 31 02:55:25.501463 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:25.501433 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1181c942_8297_48bf_b860_16d8dd8f6a8a.slice/crio-dd144bc54bf83edd53004acbaddcd14fab35ea101ba97f7c05b0e7239427e8b0 WatchSource:0}: Error finding container dd144bc54bf83edd53004acbaddcd14fab35ea101ba97f7c05b0e7239427e8b0: Status 404 returned error can't find the container with id dd144bc54bf83edd53004acbaddcd14fab35ea101ba97f7c05b0e7239427e8b0 Oct 31 02:55:25.512149 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.512096 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qlrx" Oct 31 02:55:25.514969 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.514948 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj"] Oct 31 02:55:25.518444 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:25.518425 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb2947c_46ed_4f8d_a837_d1004e3ac459.slice/crio-a4b9dea38c7460b01e40927ddb33f36202f6caee3a59b4de045e255ced45cff9 WatchSource:0}: Error finding container a4b9dea38c7460b01e40927ddb33f36202f6caee3a59b4de045e255ced45cff9: Status 404 returned error can't find the container with id a4b9dea38c7460b01e40927ddb33f36202f6caee3a59b4de045e255ced45cff9 Oct 31 02:55:25.538332 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.537655 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" event={"ID":"1181c942-8297-48bf-b860-16d8dd8f6a8a","Type":"ContainerStarted","Data":"dd144bc54bf83edd53004acbaddcd14fab35ea101ba97f7c05b0e7239427e8b0"} Oct 31 02:55:25.540726 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.540533 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" event={"ID":"8eb2947c-46ed-4f8d-a837-d1004e3ac459","Type":"ContainerStarted","Data":"a4b9dea38c7460b01e40927ddb33f36202f6caee3a59b4de045e255ced45cff9"} Oct 31 02:55:25.790272 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:25.790170 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qlrx"] Oct 31 02:55:25.794154 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:25.794128 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111372d5_9b74_4bae_b96f_3671787ad5b3.slice/crio-71d7d17345c673f18d27c2570870da5ccabc91aa03a5ec3a10641dff341ee919 WatchSource:0}: Error finding container 71d7d17345c673f18d27c2570870da5ccabc91aa03a5ec3a10641dff341ee919: Status 404 returned error can't find the container with id 71d7d17345c673f18d27c2570870da5ccabc91aa03a5ec3a10641dff341ee919 Oct 31 02:55:26.545603 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:26.545569 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" event={"ID":"1181c942-8297-48bf-b860-16d8dd8f6a8a","Type":"ContainerStarted","Data":"5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3"} Oct 31 02:55:26.546930 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:26.546903 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qlrx" event={"ID":"111372d5-9b74-4bae-b96f-3671787ad5b3","Type":"ContainerStarted","Data":"71d7d17345c673f18d27c2570870da5ccabc91aa03a5ec3a10641dff341ee919"} Oct 31 02:55:26.548426 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:26.548367 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" event={"ID":"aca0a795-839c-4055-b009-793e1ffc4d58","Type":"ContainerStarted","Data":"54a5f65d4b551bbca7f419f5b38d48d3d5d11e8764de8815de9cc1098db8161e"} Oct 31 02:55:26.550101 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:26.550082 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" event={"ID":"8eb2947c-46ed-4f8d-a837-d1004e3ac459","Type":"ContainerStarted","Data":"0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb"} Oct 31 02:55:26.692738 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:26.692703 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:26.692998 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:26.692977 4854 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:27.555913 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:27.555835 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qlrx" event={"ID":"111372d5-9b74-4bae-b96f-3671787ad5b3","Type":"ContainerStarted","Data":"fe5e33103cb44c3d95af5f0eb47d83a3317f0007b9a4a2eed1d243a9ed5a1bdf"} Oct 31 02:55:27.555913 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:27.555900 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qlrx" event={"ID":"111372d5-9b74-4bae-b96f-3671787ad5b3","Type":"ContainerStarted","Data":"e8cca3a85ba0991e3db0b8af2a43d857cb063e6509ecf9749a4708390bedd3ca"} Oct 31 02:55:27.557773 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:27.557752 4854 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:27.557773 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:27.557764 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" event={"ID":"aca0a795-839c-4055-b009-793e1ffc4d58","Type":"ContainerStarted","Data":"3ce818ae2776c3cd8804a5dcd5898430f7cc0440974ad4cd70cc3c4ba12f313c"} Oct 31 02:55:27.557941 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:27.557807 4854 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:27.557941 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:27.557821 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8qqdp" event={"ID":"aca0a795-839c-4055-b009-793e1ffc4d58","Type":"ContainerStarted","Data":"a1ab9afc212bd93d1738af10e53cd19bbb7cc1b12d5d05aca78510da7ca33aea"} Oct 31 02:55:28.449052 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.449016 4854 scope.go:122] "RemoveContainer" containerID="e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38" Oct 31 02:55:28.562200 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.562179 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 02:55:28.562663 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.562649 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/2.log" Oct 31 02:55:28.562714 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.562681 4854 generic.go:356] "Generic (PLEG): container finished" podID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" containerID="76fc3853937a33daa1e654df47744ff0e56095f3bdb1df78bef4d5cd4bb8242f" exitCode=255 Oct 31 02:55:28.562773 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.562754 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" event={"ID":"ec216fff-d06e-4a0a-b35e-73b3a2d513a8","Type":"ContainerDied","Data":"76fc3853937a33daa1e654df47744ff0e56095f3bdb1df78bef4d5cd4bb8242f"} Oct 31 02:55:28.562823 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.562812 4854 scope.go:122] "RemoveContainer" containerID="e5d16ce4d37789f95dde287d8d958055a7de564be70b84bff90b0378aa6ebe38" Oct 31 02:55:28.563739 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:28.563714 4854 scope.go:122] "RemoveContainer" containerID="76fc3853937a33daa1e654df47744ff0e56095f3bdb1df78bef4d5cd4bb8242f" Oct 31 02:55:28.564090 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:28.564059 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:29.567488 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:29.567462 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 02:55:29.702333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:29.702305 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:30.559612 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:30.559580 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:31.600742 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:31.600712 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:31.601168 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:31.601038 4854 scope.go:122] "RemoveContainer" containerID="76fc3853937a33daa1e654df47744ff0e56095f3bdb1df78bef4d5cd4bb8242f" Oct 31 02:55:31.601297 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:31.601273 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:33.183439 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.183402 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:33.183832 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.183475 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:33.184000 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.183977 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfdd5bc7-bedb-4abc-96ec-9e7d9f803702-service-ca-bundle\") pod \"router-default-66695d6bd7-5pk65\" (UID: \"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702\") " pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:33.186038 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.186018 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48eda75e-e883-4929-8278-fd49da0f5b27-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5k4sc\" (UID: \"48eda75e-e883-4929-8278-fd49da0f5b27\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:33.284040 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.284017 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:33.284173 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.284059 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:33.286431 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.286405 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84351d2c-2829-4e33-afa0-0f3f813dd4fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rsvkk\" (UID: \"84351d2c-2829-4e33-afa0-0f3f813dd4fa\") " pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:33.286557 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.286534 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3be8a54-63a4-408c-ae91-e21314a80325-metrics-tls\") pod \"dns-default-pwpgn\" (UID: \"b3be8a54-63a4-408c-ae91-e21314a80325\") " pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:33.308416 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.308395 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:33.311607 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.311586 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" Oct 31 02:55:33.320678 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.320619 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:33.323784 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.323765 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rsvkk" Oct 31 02:55:33.595030 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.595001 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5k4sc"] Oct 31 02:55:33.598597 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:33.598571 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48eda75e_e883_4929_8278_fd49da0f5b27.slice/crio-9b4504fa863271cee4863c9c361e4f0b820d5da332438be0c5ff07715d480ef9 WatchSource:0}: Error finding container 9b4504fa863271cee4863c9c361e4f0b820d5da332438be0c5ff07715d480ef9: Status 404 returned error can't find the container with id 9b4504fa863271cee4863c9c361e4f0b820d5da332438be0c5ff07715d480ef9 Oct 31 02:55:33.602997 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.602969 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66695d6bd7-5pk65"] Oct 31 02:55:33.607559 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:33.605098 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdd5bc7_bedb_4abc_96ec_9e7d9f803702.slice/crio-21738838faf900c9a4973944c105349722b6438f679ba15c364c8f422ea25963 WatchSource:0}: Error finding container 21738838faf900c9a4973944c105349722b6438f679ba15c364c8f422ea25963: Status 404 returned error can't find the container with id 21738838faf900c9a4973944c105349722b6438f679ba15c364c8f422ea25963 Oct 31 02:55:33.623773 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.623744 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rsvkk"] Oct 31 02:55:33.629044 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.628987 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwpgn"] Oct 31 02:55:33.631331 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:33.631308 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84351d2c_2829_4e33_afa0_0f3f813dd4fa.slice/crio-55c23002635737bc4f7974bf401366649b1fcc2ebb86ce9c79ca92b9900ff9f6 WatchSource:0}: Error finding container 55c23002635737bc4f7974bf401366649b1fcc2ebb86ce9c79ca92b9900ff9f6: Status 404 returned error can't find the container with id 55c23002635737bc4f7974bf401366649b1fcc2ebb86ce9c79ca92b9900ff9f6 Oct 31 02:55:33.632161 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:33.632136 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3be8a54_63a4_408c_ae91_e21314a80325.slice/crio-755af960fe040a619139ec09159f589344b9809eb1a6a0e39095523e811b64a8 WatchSource:0}: Error finding container 755af960fe040a619139ec09159f589344b9809eb1a6a0e39095523e811b64a8: Status 404 returned error can't find the container with id 755af960fe040a619139ec09159f589344b9809eb1a6a0e39095523e811b64a8 Oct 31 02:55:33.853461 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.853387 4854 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:33.853784 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:33.853763 4854 scope.go:122] "RemoveContainer" containerID="76fc3853937a33daa1e654df47744ff0e56095f3bdb1df78bef4d5cd4bb8242f" Oct 31 02:55:33.854001 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:33.853972 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8j7j_openshift-console-operator(ec216fff-d06e-4a0a-b35e-73b3a2d513a8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" podUID="ec216fff-d06e-4a0a-b35e-73b3a2d513a8" Oct 31 02:55:34.589034 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.588995 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsvkk" event={"ID":"84351d2c-2829-4e33-afa0-0f3f813dd4fa","Type":"ContainerStarted","Data":"c42c54cf748abd05615c6102aa53a0604c7e48446755175d240462dfd2afd4c6"} Oct 31 02:55:34.589509 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.589064 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsvkk" event={"ID":"84351d2c-2829-4e33-afa0-0f3f813dd4fa","Type":"ContainerStarted","Data":"ea7b197bfe287c664314e7c15b29f8e01ce614578141c1d9fc37d9bcbf694be3"} Oct 31 02:55:34.589509 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.589094 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsvkk" event={"ID":"84351d2c-2829-4e33-afa0-0f3f813dd4fa","Type":"ContainerStarted","Data":"55c23002635737bc4f7974bf401366649b1fcc2ebb86ce9c79ca92b9900ff9f6"} Oct 31 02:55:34.590782 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.590750 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" event={"ID":"48eda75e-e883-4929-8278-fd49da0f5b27","Type":"ContainerStarted","Data":"9b4504fa863271cee4863c9c361e4f0b820d5da332438be0c5ff07715d480ef9"} Oct 31 02:55:34.592774 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.592744 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66695d6bd7-5pk65" event={"ID":"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702","Type":"ContainerStarted","Data":"b605429be974244b784f4a19a89bc12b37be9f874048222017642a70214cae01"} Oct 31 02:55:34.592905 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.592793 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66695d6bd7-5pk65" event={"ID":"bfdd5bc7-bedb-4abc-96ec-9e7d9f803702","Type":"ContainerStarted","Data":"21738838faf900c9a4973944c105349722b6438f679ba15c364c8f422ea25963"} Oct 31 02:55:34.594986 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:34.594965 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwpgn" event={"ID":"b3be8a54-63a4-408c-ae91-e21314a80325","Type":"ContainerStarted","Data":"755af960fe040a619139ec09159f589344b9809eb1a6a0e39095523e811b64a8"} Oct 31 02:55:35.309480 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:35.309411 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:35.312402 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:35.312369 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:35.605754 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:35.605714 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5k4sc" event={"ID":"48eda75e-e883-4929-8278-fd49da0f5b27","Type":"ContainerStarted","Data":"bbf66dc8012fc769609a176d7e9749d70351a159c58f99f76b538451f55e641f"} Oct 31 02:55:35.608182 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:35.607631 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwpgn" event={"ID":"b3be8a54-63a4-408c-ae91-e21314a80325","Type":"ContainerStarted","Data":"c78b2fcb27550b138ba9406488b76810b7543fcde7f922bf123214a46eff0d50"} Oct 31 02:55:35.608182 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:35.608045 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:35.609273 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:35.609143 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66695d6bd7-5pk65" Oct 31 02:55:36.613110 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:36.613075 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwpgn" event={"ID":"b3be8a54-63a4-408c-ae91-e21314a80325","Type":"ContainerStarted","Data":"e106ae41d8e4272a7e8d8feb5755a13f7c833c7164587fcff6f5a5951befa634"} Oct 31 02:55:36.613605 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:36.613231 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:36.615308 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:36.615272 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rsvkk" event={"ID":"84351d2c-2829-4e33-afa0-0f3f813dd4fa","Type":"ContainerStarted","Data":"605b4c33deecca43257c22c1420d4ef60d35f1770fe225fe771c1a0b6fbaf97a"} Oct 31 02:55:36.852529 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:36.852492 4854 patch_prober.go:29] interesting pod/image-registry-7d8c6f6dc9-zfrlj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 02:55:36.852678 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:36.852559 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:55:39.025163 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:39.025133 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75b45ffb98-wkq9k"] Oct 31 02:55:39.249847 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:55:39.249813 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:55:40.564529 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:40.564496 4854 patch_prober.go:29] interesting pod/image-registry-7d8c6f6dc9-zfrlj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 02:55:40.564929 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:40.564545 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:55:45.235294 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:45.235263 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5nsnk" Oct 31 02:55:46.287071 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.287032 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xlk6q"] Oct 31 02:55:46.331891 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.331862 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.386425 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.386387 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 02:55:46.386562 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.386506 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 02:55:46.386606 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.386577 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtdnh\"" Oct 31 02:55:46.418620 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.418598 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 02:55:46.418758 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.418678 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 02:55:46.469077 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469053 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4dc82-af0b-49a3-b398-b9a7caafd206-metrics-client-ca\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469189 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469093 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwt4\" (UniqueName: \"kubernetes.io/projected/e1e4dc82-af0b-49a3-b398-b9a7caafd206-kube-api-access-gmwt4\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469189 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469117 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-textfile\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469189 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469137 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-tls\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469315 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469289 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-sys\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469352 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469318 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-wtmp\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469352 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469336 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469414 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469402 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-root\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.469449 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.469436 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570143 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570081 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570143 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570126 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4dc82-af0b-49a3-b398-b9a7caafd206-metrics-client-ca\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570153 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwt4\" (UniqueName: \"kubernetes.io/projected/e1e4dc82-af0b-49a3-b398-b9a7caafd206-kube-api-access-gmwt4\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570177 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-textfile\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570203 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-tls\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570266 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-sys\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570293 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-wtmp\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570315 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570641 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570342 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-root\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570641 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570416 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-root\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570641 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570464 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-sys\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570641 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570499 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-wtmp\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570641 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570541 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-textfile\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570812 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570712 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.570861 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.570841 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4dc82-af0b-49a3-b398-b9a7caafd206-metrics-client-ca\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.572632 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.572610 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.572714 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.572704 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1e4dc82-af0b-49a3-b398-b9a7caafd206-node-exporter-tls\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.582033 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.582013 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwt4\" (UniqueName: \"kubernetes.io/projected/e1e4dc82-af0b-49a3-b398-b9a7caafd206-kube-api-access-gmwt4\") pod \"node-exporter-xlk6q\" (UID: \"e1e4dc82-af0b-49a3-b398-b9a7caafd206\") " pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.620703 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.620677 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pwpgn" Oct 31 02:55:46.681671 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:46.681645 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xlk6q" Oct 31 02:55:46.721130 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:55:46.721105 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e4dc82_af0b_49a3_b398_b9a7caafd206.slice/crio-617e37b28f1d0b1ee145d45085a75d45ccf0984172b596cbb0b6139ddcb44ff0 WatchSource:0}: Error finding container 617e37b28f1d0b1ee145d45085a75d45ccf0984172b596cbb0b6139ddcb44ff0: Status 404 returned error can't find the container with id 617e37b28f1d0b1ee145d45085a75d45ccf0984172b596cbb0b6139ddcb44ff0 Oct 31 02:55:47.658145 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:47.658106 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlk6q" event={"ID":"e1e4dc82-af0b-49a3-b398-b9a7caafd206","Type":"ContainerStarted","Data":"617e37b28f1d0b1ee145d45085a75d45ccf0984172b596cbb0b6139ddcb44ff0"} Oct 31 02:55:48.663302 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:48.663215 4854 generic.go:356] "Generic (PLEG): container finished" podID="e1e4dc82-af0b-49a3-b398-b9a7caafd206" containerID="dd9d36e207bec136fd1bd89f076779b17478204d09bd9a8dfd25b9bf698adbd5" exitCode=0 Oct 31 02:55:48.663635 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:48.663295 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlk6q" event={"ID":"e1e4dc82-af0b-49a3-b398-b9a7caafd206","Type":"ContainerDied","Data":"dd9d36e207bec136fd1bd89f076779b17478204d09bd9a8dfd25b9bf698adbd5"} Oct 31 02:55:49.030416 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:49.030393 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:55:49.669188 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:49.669149 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlk6q" event={"ID":"e1e4dc82-af0b-49a3-b398-b9a7caafd206","Type":"ContainerStarted","Data":"cfbb4b2ec60998cfeb3bae902245ba6b4104586a98817c99f655a99e51a92e44"} Oct 31 02:55:49.669568 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:49.669210 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlk6q" event={"ID":"e1e4dc82-af0b-49a3-b398-b9a7caafd206","Type":"ContainerStarted","Data":"7cf1d4df954cecc346a62a248d6295e0e52fcfe1bcce8337796d6ef96946e4dc"} Oct 31 02:55:49.688961 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:49.688912 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xlk6q" podStartSLOduration=2.043752124 podStartE2EDuration="3.688898347s" totalImagesPullingTime="1.555377074s" totalInitContainerRuntime="89.769149ms" isStatefulPod=true podCreationTimestamp="2026-10-31 02:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:49.68783969 +0000 UTC m=+33.385007122" watchObservedRunningTime="2026-10-31 02:55:49.688898347 +0000 UTC m=+33.386065650" Oct 31 02:55:50.564067 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:50.564035 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:55:51.034038 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:51.034011 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r76hd" Oct 31 02:55:52.449408 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:52.449376 4854 scope.go:122] "RemoveContainer" containerID="76fc3853937a33daa1e654df47744ff0e56095f3bdb1df78bef4d5cd4bb8242f" Oct 31 02:55:52.682689 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:52.682664 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 02:55:52.682838 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:52.682781 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" event={"ID":"ec216fff-d06e-4a0a-b35e-73b3a2d513a8","Type":"ContainerStarted","Data":"0b5da6ab266f9e6fe0aec2610283e1dd37390f571a13ecda0b8fef4c3b4554af"} Oct 31 02:55:52.683068 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:52.683052 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:55:52.780868 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:55:52.780846 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8j7j" Oct 31 02:56:01.422482 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:01.422408 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj"] Oct 31 02:56:04.049120 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.049075 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" podUID="1181c942-8297-48bf-b860-16d8dd8f6a8a" containerName="registry" containerID="cri-o://5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3" gracePeriod=30 Oct 31 02:56:04.300456 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.300405 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:56:04.310384 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310357 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310493 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310398 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310493 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310422 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310493 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310452 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310493 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310490 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310652 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310535 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310652 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310563 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310652 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310602 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") pod \"1181c942-8297-48bf-b860-16d8dd8f6a8a\" (UID: \"1181c942-8297-48bf-b860-16d8dd8f6a8a\") " Oct 31 02:56:04.310989 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310903 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:04.310989 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.310926 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:04.313300 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.313208 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:04.313516 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.313474 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:04.313658 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.313606 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "kube-api-access-brs2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:04.313658 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.313606 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:04.313742 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.313691 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:04.317963 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.317940 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted" pod "1181c942-8297-48bf-b860-16d8dd8f6a8a" (UID: "1181c942-8297-48bf-b860-16d8dd8f6a8a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:04.411988 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.411955 4854 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.411988 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.411982 4854 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1181c942-8297-48bf-b860-16d8dd8f6a8a-ca-trust-extracted\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.411988 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.411991 4854 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-bound-sa-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.412210 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.412000 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brs2h\" (UniqueName: \"kubernetes.io/projected/1181c942-8297-48bf-b860-16d8dd8f6a8a-kube-api-access-brs2h\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.412210 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.412010 4854 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-trusted-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.412210 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.412018 4854 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1181c942-8297-48bf-b860-16d8dd8f6a8a-registry-certificates\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.412210 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.412052 4854 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-image-registry-private-configuration\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.412210 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.412062 4854 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1181c942-8297-48bf-b860-16d8dd8f6a8a-installation-pull-secrets\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:04.734498 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.734466 4854 generic.go:356] "Generic (PLEG): container finished" podID="1181c942-8297-48bf-b860-16d8dd8f6a8a" containerID="5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3" exitCode=0 Oct 31 02:56:04.734676 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.734513 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" Oct 31 02:56:04.734676 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.734520 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" event={"ID":"1181c942-8297-48bf-b860-16d8dd8f6a8a","Type":"ContainerDied","Data":"5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3"} Oct 31 02:56:04.734676 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.734571 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b45ffb98-wkq9k" event={"ID":"1181c942-8297-48bf-b860-16d8dd8f6a8a","Type":"ContainerDied","Data":"dd144bc54bf83edd53004acbaddcd14fab35ea101ba97f7c05b0e7239427e8b0"} Oct 31 02:56:04.734676 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.734597 4854 scope.go:122] "RemoveContainer" containerID="5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3" Oct 31 02:56:04.742749 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.742732 4854 scope.go:122] "RemoveContainer" containerID="5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3" Oct 31 02:56:04.743037 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:56:04.743015 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3\": container with ID starting with 5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3 not found: ID does not exist" containerID="5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3" Oct 31 02:56:04.743086 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.743046 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3"} err="failed to get container status \"5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3\": rpc error: code = NotFound desc = could not find container \"5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3\": container with ID starting with 5416e48fe11eacc2923b580c0171dfac7ca836b69e8905ca305a5418a94f13b3 not found: ID does not exist" Oct 31 02:56:04.748714 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.748695 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75b45ffb98-wkq9k"] Oct 31 02:56:04.754914 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:04.754892 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75b45ffb98-wkq9k"] Oct 31 02:56:06.453425 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:06.453393 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1181c942-8297-48bf-b860-16d8dd8f6a8a" path="/var/lib/kubelet/pods/1181c942-8297-48bf-b860-16d8dd8f6a8a/volumes" Oct 31 02:56:09.249675 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:56:09.249626 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:56:24.811005 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:24.810942 4854 generic.go:356] "Generic (PLEG): container finished" podID="e95878f0-c838-42c0-a546-55f075727d45" containerID="f69f3dba55a7674deb6afe025f01c2cfddbf8d6cd91335ddd4af15bcbc459b07" exitCode=0 Oct 31 02:56:24.811468 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:24.811021 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" event={"ID":"e95878f0-c838-42c0-a546-55f075727d45","Type":"ContainerDied","Data":"f69f3dba55a7674deb6afe025f01c2cfddbf8d6cd91335ddd4af15bcbc459b07"} Oct 31 02:56:24.811468 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:24.811390 4854 scope.go:122] "RemoveContainer" containerID="f69f3dba55a7674deb6afe025f01c2cfddbf8d6cd91335ddd4af15bcbc459b07" Oct 31 02:56:25.816239 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:25.816205 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5vq46" event={"ID":"e95878f0-c838-42c0-a546-55f075727d45","Type":"ContainerStarted","Data":"b8d0f60f32723e8b0b6490229d10f86ac702ea7acdd1e3acb30f6e4df1835f55"} Oct 31 02:56:26.442539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.442467 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerName="registry" containerID="cri-o://0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb" gracePeriod=30 Oct 31 02:56:26.676847 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.676823 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:56:26.770686 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770659 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.770814 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770701 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.770814 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770734 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.770814 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770756 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.770814 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770774 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.770814 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770799 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.771063 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.770823 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.771126 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.771065 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") pod \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\" (UID: \"8eb2947c-46ed-4f8d-a837-d1004e3ac459\") " Oct 31 02:56:26.771276 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.771234 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:26.771356 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.771274 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:26.771394 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.771382 4854 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-trusted-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.771429 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.771404 4854 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-certificates\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.773057 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.773013 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:26.773144 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.773096 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:26.773709 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.773680 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:26.773785 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.773686 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:26.773843 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.773801 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "kube-api-access-jlxrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:26.778004 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.777982 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted" pod "8eb2947c-46ed-4f8d-a837-d1004e3ac459" (UID: "8eb2947c-46ed-4f8d-a837-d1004e3ac459"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:26.820676 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.820647 4854 generic.go:356] "Generic (PLEG): container finished" podID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerID="0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb" exitCode=0 Oct 31 02:56:26.820975 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.820695 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" Oct 31 02:56:26.820975 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.820724 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" event={"ID":"8eb2947c-46ed-4f8d-a837-d1004e3ac459","Type":"ContainerDied","Data":"0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb"} Oct 31 02:56:26.820975 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.820775 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj" event={"ID":"8eb2947c-46ed-4f8d-a837-d1004e3ac459","Type":"ContainerDied","Data":"a4b9dea38c7460b01e40927ddb33f36202f6caee3a59b4de045e255ced45cff9"} Oct 31 02:56:26.820975 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.820795 4854 scope.go:122] "RemoveContainer" containerID="0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb" Oct 31 02:56:26.829078 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.829061 4854 scope.go:122] "RemoveContainer" containerID="0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb" Oct 31 02:56:26.829364 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:56:26.829347 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb\": container with ID starting with 0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb not found: ID does not exist" containerID="0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb" Oct 31 02:56:26.829422 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.829372 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb"} err="failed to get container status \"0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb\": rpc error: code = NotFound desc = could not find container \"0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb\": container with ID starting with 0c97f9b80000220e739eaf07229555f33d583927e70123610879e9b59fdc11fb not found: ID does not exist" Oct 31 02:56:26.839835 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.839815 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj"] Oct 31 02:56:26.844796 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.844776 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d8c6f6dc9-zfrlj"] Oct 31 02:56:26.872428 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.872402 4854 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-bound-sa-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.872428 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.872422 4854 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8eb2947c-46ed-4f8d-a837-d1004e3ac459-ca-trust-extracted\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.872537 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.872434 4854 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-installation-pull-secrets\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.872537 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.872443 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlxrw\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-kube-api-access-jlxrw\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.872537 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.872452 4854 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8eb2947c-46ed-4f8d-a837-d1004e3ac459-registry-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:26.872537 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:26.872461 4854 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8eb2947c-46ed-4f8d-a837-d1004e3ac459-image-registry-private-configuration\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:28.453553 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:28.453520 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" path="/var/lib/kubelet/pods/8eb2947c-46ed-4f8d-a837-d1004e3ac459/volumes" Oct 31 02:56:37.866747 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:37.866710 4854 generic.go:356] "Generic (PLEG): container finished" podID="32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8" containerID="e38e12c9832fefe2d5d24a47cb0abd4e99e758d1540da31c35af53a538f22b3c" exitCode=0 Oct 31 02:56:37.867332 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:37.866784 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" event={"ID":"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8","Type":"ContainerDied","Data":"e38e12c9832fefe2d5d24a47cb0abd4e99e758d1540da31c35af53a538f22b3c"} Oct 31 02:56:37.867332 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:37.867136 4854 scope.go:122] "RemoveContainer" containerID="e38e12c9832fefe2d5d24a47cb0abd4e99e758d1540da31c35af53a538f22b3c" Oct 31 02:56:37.868674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:37.868574 4854 generic.go:356] "Generic (PLEG): container finished" podID="e7194787-0fa9-43a4-a749-c6aac9433cfc" containerID="dfba916d4f8584000e1b335f54b8294243be242bf9f5357020641cfcafa64af4" exitCode=0 Oct 31 02:56:37.868674 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:37.868646 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xth74" event={"ID":"e7194787-0fa9-43a4-a749-c6aac9433cfc","Type":"ContainerDied","Data":"dfba916d4f8584000e1b335f54b8294243be242bf9f5357020641cfcafa64af4"} Oct 31 02:56:37.868959 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:37.868943 4854 scope.go:122] "RemoveContainer" containerID="dfba916d4f8584000e1b335f54b8294243be242bf9f5357020641cfcafa64af4" Oct 31 02:56:38.270987 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.270953 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wwvvn-76z5v"] Oct 31 02:56:38.271324 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.271309 4854 cpu_manager.go:414] "containerMap: removing container" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerName="registry" Oct 31 02:56:38.271397 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.271325 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerName="registry" Oct 31 02:56:38.271397 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.271337 4854 cpu_manager.go:414] "containerMap: removing container" podUID="1181c942-8297-48bf-b860-16d8dd8f6a8a" containerName="registry" Oct 31 02:56:38.271397 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.271341 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1181c942-8297-48bf-b860-16d8dd8f6a8a" containerName="registry" Oct 31 02:56:38.271539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.271412 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb2947c-46ed-4f8d-a837-d1004e3ac459" containerName="registry" Oct 31 02:56:38.271539 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.271420 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="1181c942-8297-48bf-b860-16d8dd8f6a8a" containerName="registry" Oct 31 02:56:38.274407 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.274390 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.277289 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.277272 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w5s96\"" Oct 31 02:56:38.284490 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.284470 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wwvvn-76z5v"] Oct 31 02:56:38.359684 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.359656 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.359838 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.359699 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.359907 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.359842 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.359907 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.359887 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpr7\" (UniqueName: \"kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.461168 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.461146 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.461335 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.461182 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpr7\" (UniqueName: \"kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.461335 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.461224 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.461335 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.461289 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.461600 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.461580 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.461828 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.461811 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.462222 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.462204 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.473369 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.473352 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpr7\" (UniqueName: \"kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7\") pod \"periodic-gathering-wwvvn-76z5v\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.586491 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.586440 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:56:38.857333 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.857234 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wwvvn-76z5v"] Oct 31 02:56:38.860510 ip-10-0-130-12 kubenswrapper[4854]: W1031 02:56:38.860477 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbff375c_6137_4767_b8e8_d8aa057f3e77.slice/crio-7ba040ea94135dc31da5a924b1d488f675f8df95925a1bc0bd0c90113ef3bae0 WatchSource:0}: Error finding container 7ba040ea94135dc31da5a924b1d488f675f8df95925a1bc0bd0c90113ef3bae0: Status 404 returned error can't find the container with id 7ba040ea94135dc31da5a924b1d488f675f8df95925a1bc0bd0c90113ef3bae0 Oct 31 02:56:38.880056 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.880025 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-q6xhf" event={"ID":"32eb238c-4f7d-47ec-8e6a-3fe59bdbdea8","Type":"ContainerStarted","Data":"147b48d76e3a66c265e3b306bbcf1230ae727f527e57c6947b6445c56fab9887"} Oct 31 02:56:38.882052 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.882028 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xth74" event={"ID":"e7194787-0fa9-43a4-a749-c6aac9433cfc","Type":"ContainerStarted","Data":"dc31c16d7cde6f1be022f78452f061c7b9f21304796ec410e279f0d7793e3a2e"} Oct 31 02:56:38.883224 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:38.883202 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" event={"ID":"cbff375c-6137-4767-b8e8-d8aa057f3e77","Type":"ContainerStarted","Data":"7ba040ea94135dc31da5a924b1d488f675f8df95925a1bc0bd0c90113ef3bae0"} Oct 31 02:56:39.249212 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:56:39.249180 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:56:39.887946 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:39.887913 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" event={"ID":"cbff375c-6137-4767-b8e8-d8aa057f3e77","Type":"ContainerStarted","Data":"47780f567cb524a3cc912638a0d22c019a1265f9024e0154e485eaf647c79d3a"} Oct 31 02:56:39.904582 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:56:39.904533 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" podStartSLOduration=1.9045199560000001 podStartE2EDuration="1.904519956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:39.903913639 +0000 UTC m=+83.601080940" watchObservedRunningTime="2026-10-31 02:56:39.904519956 +0000 UTC m=+83.601687251" Oct 31 02:57:09.251386 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:57:09.251331 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:57:39.250819 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:57:39.250732 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:58:02.195329 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:02.195300 4854 generic.go:356] "Generic (PLEG): container finished" podID="cbff375c-6137-4767-b8e8-d8aa057f3e77" containerID="47780f567cb524a3cc912638a0d22c019a1265f9024e0154e485eaf647c79d3a" exitCode=0 Oct 31 02:58:02.195703 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:02.195373 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" event={"ID":"cbff375c-6137-4767-b8e8-d8aa057f3e77","Type":"ContainerDied","Data":"47780f567cb524a3cc912638a0d22c019a1265f9024e0154e485eaf647c79d3a"} Oct 31 02:58:03.341010 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.340987 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:58:03.371180 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.371158 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path\") pod \"cbff375c-6137-4767-b8e8-d8aa057f3e77\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " Oct 31 02:58:03.371339 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.371191 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7\" (UniqueName: \"kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7\") pod \"cbff375c-6137-4767-b8e8-d8aa057f3e77\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " Oct 31 02:58:03.371339 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.371213 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle\") pod \"cbff375c-6137-4767-b8e8-d8aa057f3e77\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " Oct 31 02:58:03.371339 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.371286 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle\") pod \"cbff375c-6137-4767-b8e8-d8aa057f3e77\" (UID: \"cbff375c-6137-4767-b8e8-d8aa057f3e77\") " Oct 31 02:58:03.371717 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.371687 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle" pod "cbff375c-6137-4767-b8e8-d8aa057f3e77" (UID: "cbff375c-6137-4767-b8e8-d8aa057f3e77"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:58:03.371877 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.371690 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle" pod "cbff375c-6137-4767-b8e8-d8aa057f3e77" (UID: "cbff375c-6137-4767-b8e8-d8aa057f3e77"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:58:03.372137 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.372115 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path" pod "cbff375c-6137-4767-b8e8-d8aa057f3e77" (UID: "cbff375c-6137-4767-b8e8-d8aa057f3e77"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:58:03.373635 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.373610 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7" pod "cbff375c-6137-4767-b8e8-d8aa057f3e77" (UID: "cbff375c-6137-4767-b8e8-d8aa057f3e77"). InnerVolumeSpecName "kube-api-access-chpr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:58:03.472385 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.472339 4854 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-trusted-ca-bundle\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:03.472385 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.472358 4854 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbff375c-6137-4767-b8e8-d8aa057f3e77-service-ca-bundle\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:03.472385 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.472369 4854 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cbff375c-6137-4767-b8e8-d8aa057f3e77-archives-path\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:03.472385 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:03.472378 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chpr7\" (UniqueName: \"kubernetes.io/projected/cbff375c-6137-4767-b8e8-d8aa057f3e77-kube-api-access-chpr7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:58:04.203527 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:04.203490 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" event={"ID":"cbff375c-6137-4767-b8e8-d8aa057f3e77","Type":"ContainerDied","Data":"7ba040ea94135dc31da5a924b1d488f675f8df95925a1bc0bd0c90113ef3bae0"} Oct 31 02:58:04.203527 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:04.203529 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwvvn-76z5v" Oct 31 02:58:04.203729 ip-10-0-130-12 kubenswrapper[4854]: I1031 02:58:04.203544 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba040ea94135dc31da5a924b1d488f675f8df95925a1bc0bd0c90113ef3bae0" Oct 31 02:58:09.250014 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:58:09.249973 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:58:39.251240 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:58:39.251201 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:59:09.252075 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:59:09.252032 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 02:59:39.252426 ip-10-0-130-12 kubenswrapper[4854]: E1031 02:59:39.252387 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:00:09.252285 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:00:09.252221 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:00:10.428790 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.428610 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk"] Oct 31 03:00:10.429262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.428973 4854 cpu_manager.go:414] "containerMap: removing container" podUID="cbff375c-6137-4767-b8e8-d8aa057f3e77" containerName="insights-gathering" Oct 31 03:00:10.429262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.428986 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbff375c-6137-4767-b8e8-d8aa057f3e77" containerName="insights-gathering" Oct 31 03:00:10.429262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.429045 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbff375c-6137-4767-b8e8-d8aa057f3e77" containerName="insights-gathering" Oct 31 03:00:10.431891 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.431876 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.435116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.435098 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:00:10.438285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.438220 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:00:10.438447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.438408 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:00:10.438447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.438431 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:00:10.438574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.438525 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-97phk\"" Oct 31 03:00:10.464807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.464780 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk"] Oct 31 03:00:10.524629 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.524598 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.524760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.524633 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.524760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.524664 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.524845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.524819 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.524901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.524873 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2hf\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.525033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.525007 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.525106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.525075 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.625633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625609 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2hf\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.625753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625639 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.625753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625658 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.625753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625695 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.625753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625720 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.625753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625751 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.626033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.625780 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.626507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.626414 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.628205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.628175 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.628330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.628205 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.628376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.628331 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.628376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.628348 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.637135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.637113 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.637528 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.637505 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2hf\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf\") pod \"istiod-openshift-gateway-7876bfcb74-dxkkk\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:10.742778 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:10.742757 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:11.016444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:11.016369 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk"] Oct 31 03:00:11.019990 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:00:11.019964 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fa8120_4f33_4995_9b5e_adcfa96b3c16.slice/crio-38867c5c7b566c4740e8572d84fce2eae9ff18524a14779f8635d0871d24b187 WatchSource:0}: Error finding container 38867c5c7b566c4740e8572d84fce2eae9ff18524a14779f8635d0871d24b187: Status 404 returned error can't find the container with id 38867c5c7b566c4740e8572d84fce2eae9ff18524a14779f8635d0871d24b187 Oct 31 03:00:11.660802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:11.660770 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" event={"ID":"e5fa8120-4f33-4995-9b5e-adcfa96b3c16","Type":"ContainerStarted","Data":"38867c5c7b566c4740e8572d84fce2eae9ff18524a14779f8635d0871d24b187"} Oct 31 03:00:15.513611 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:15.513581 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:00:15.513905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:15.513643 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:00:15.686164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:15.686130 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" event={"ID":"e5fa8120-4f33-4995-9b5e-adcfa96b3c16","Type":"ContainerStarted","Data":"5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2"} Oct 31 03:00:15.686363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:15.686191 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:15.727013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:15.726966 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" podStartSLOduration=1.23610686 podStartE2EDuration="5.726951176s" totalImagesPullingTime="4.490844316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:00:15.72670093 +0000 UTC m=+299.423868233" watchObservedRunningTime="2026-10-31 03:00:15.726951176 +0000 UTC m=+299.424118481" Oct 31 03:00:16.458150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:16.458119 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:00:16.458698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:16.458680 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:00:16.690913 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:16.690851 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:00:39.249550 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:00:39.249471 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:00:59.901918 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.901882 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m"] Oct 31 03:00:59.905190 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.905174 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:00:59.958510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.958482 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m"] Oct 31 03:00:59.971941 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.971918 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:00:59.972054 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.972026 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:00:59.972139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.972123 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:00:59.972192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.972125 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:00:59.972324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.972311 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:00:59.972390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.972323 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-xc62g\"" Oct 31 03:00:59.990143 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.990117 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:00:59.990235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.990149 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb4c9659-e9be-4b54-adee-ffe1f5611604-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:00:59.990235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.990172 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:00:59.990235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.990187 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:00:59.990371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.990351 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4c9659-e9be-4b54-adee-ffe1f5611604-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:00:59.990416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:00:59.990408 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/cb4c9659-e9be-4b54-adee-ffe1f5611604-kube-api-access-pstsv\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.091585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091559 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/cb4c9659-e9be-4b54-adee-ffe1f5611604-kube-api-access-pstsv\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.091715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091591 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.091715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091615 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb4c9659-e9be-4b54-adee-ffe1f5611604-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.091715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091697 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.091883 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091731 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.091883 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091785 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4c9659-e9be-4b54-adee-ffe1f5611604-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.092000 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.091979 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb4c9659-e9be-4b54-adee-ffe1f5611604-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.094190 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.094161 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb4c9659-e9be-4b54-adee-ffe1f5611604-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.094322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.094236 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.094393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.094376 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.094449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.094411 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb4c9659-e9be-4b54-adee-ffe1f5611604-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.106418 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.106395 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/cb4c9659-e9be-4b54-adee-ffe1f5611604-kube-api-access-pstsv\") pod \"opendatahub-operator-controller-manager-57b855755-zms7m\" (UID: \"cb4c9659-e9be-4b54-adee-ffe1f5611604\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.219292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.219220 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:00.520879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.520858 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m"] Oct 31 03:01:00.525396 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:01:00.525371 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4c9659_e9be_4b54_adee_ffe1f5611604.slice/crio-9bf0bc3502650f62b752799afa2875ff7a956878c318058a63e248f8fb40797c WatchSource:0}: Error finding container 9bf0bc3502650f62b752799afa2875ff7a956878c318058a63e248f8fb40797c: Status 404 returned error can't find the container with id 9bf0bc3502650f62b752799afa2875ff7a956878c318058a63e248f8fb40797c Oct 31 03:01:00.527194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.527178 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:01:00.854321 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:00.854239 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" event={"ID":"cb4c9659-e9be-4b54-adee-ffe1f5611604","Type":"ContainerStarted","Data":"9bf0bc3502650f62b752799afa2875ff7a956878c318058a63e248f8fb40797c"} Oct 31 03:01:04.874342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:04.874264 4854 generic.go:356] "Generic (PLEG): container finished" podID="cb4c9659-e9be-4b54-adee-ffe1f5611604" containerID="336a527081bf25117b15a90c52ea2c6ea994b1f02190bed347693b9130d6d15d" exitCode=0 Oct 31 03:01:04.874342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:04.874338 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" event={"ID":"cb4c9659-e9be-4b54-adee-ffe1f5611604","Type":"ContainerDied","Data":"336a527081bf25117b15a90c52ea2c6ea994b1f02190bed347693b9130d6d15d"} Oct 31 03:01:05.880812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:05.880777 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" event={"ID":"cb4c9659-e9be-4b54-adee-ffe1f5611604","Type":"ContainerStarted","Data":"ff04346b763047ee2438cd0b76347edafc2483442cee0c22acd9ba93b4d3dead"} Oct 31 03:01:05.881189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:05.880883 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:05.949490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:05.949076 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" podStartSLOduration=2.389726207 podStartE2EDuration="6.949059489s" totalImagesPullingTime="4.120257616s" totalInitContainerRuntime="439.075666ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:05.946196866 +0000 UTC m=+349.643364170" watchObservedRunningTime="2026-10-31 03:01:05.949059489 +0000 UTC m=+349.646226792" Oct 31 03:01:09.250663 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:01:09.250622 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:01:16.887171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:16.887143 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-zms7m" Oct 31 03:01:36.863465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.863432 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29"] Oct 31 03:01:36.920482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.920451 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29"] Oct 31 03:01:36.920696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.920552 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:36.923689 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.923669 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-v76fv\"" Oct 31 03:01:36.926007 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.925985 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:01:36.926118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.926025 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:01:36.984223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:36.984193 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hc8\" (UniqueName: \"kubernetes.io/projected/763496b7-9cc7-475a-96c9-86a30d8b68c5-kube-api-access-48hc8\") pod \"limitador-operator-controller-manager-5778bb4449-rlp29\" (UID: \"763496b7-9cc7-475a-96c9-86a30d8b68c5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:37.085426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:37.085391 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hc8\" (UniqueName: \"kubernetes.io/projected/763496b7-9cc7-475a-96c9-86a30d8b68c5-kube-api-access-48hc8\") pod \"limitador-operator-controller-manager-5778bb4449-rlp29\" (UID: \"763496b7-9cc7-475a-96c9-86a30d8b68c5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:37.097106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:37.097077 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hc8\" (UniqueName: \"kubernetes.io/projected/763496b7-9cc7-475a-96c9-86a30d8b68c5-kube-api-access-48hc8\") pod \"limitador-operator-controller-manager-5778bb4449-rlp29\" (UID: \"763496b7-9cc7-475a-96c9-86a30d8b68c5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:37.246460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:37.246437 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:37.516655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:37.516581 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29"] Oct 31 03:01:37.519667 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:01:37.519632 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763496b7_9cc7_475a_96c9_86a30d8b68c5.slice/crio-efa9c1520e8ecb10116600f4aecf4288eefdf891cb27e38f1c8ef672b10a5e0f WatchSource:0}: Error finding container efa9c1520e8ecb10116600f4aecf4288eefdf891cb27e38f1c8ef672b10a5e0f: Status 404 returned error can't find the container with id efa9c1520e8ecb10116600f4aecf4288eefdf891cb27e38f1c8ef672b10a5e0f Oct 31 03:01:38.002840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.002803 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" event={"ID":"763496b7-9cc7-475a-96c9-86a30d8b68c5","Type":"ContainerStarted","Data":"efa9c1520e8ecb10116600f4aecf4288eefdf891cb27e38f1c8ef672b10a5e0f"} Oct 31 03:01:38.763882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.763845 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g"] Oct 31 03:01:38.767101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.767083 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:38.772113 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.772090 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 03:01:38.772219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.772147 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tdjqk\"" Oct 31 03:01:38.772516 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.772488 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 03:01:38.777738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.777721 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g"] Oct 31 03:01:38.901475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.901396 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvbv\" (UniqueName: \"kubernetes.io/projected/997f197c-63bc-4bfc-b84a-6e620cea33e3-kube-api-access-xxvbv\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:38.901475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.901447 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f197c-63bc-4bfc-b84a-6e620cea33e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:38.901712 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:38.901525 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/997f197c-63bc-4bfc-b84a-6e620cea33e3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.003275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.002840 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvbv\" (UniqueName: \"kubernetes.io/projected/997f197c-63bc-4bfc-b84a-6e620cea33e3-kube-api-access-xxvbv\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.003275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.003007 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f197c-63bc-4bfc-b84a-6e620cea33e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.003275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.003070 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/997f197c-63bc-4bfc-b84a-6e620cea33e3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.004058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.003910 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/997f197c-63bc-4bfc-b84a-6e620cea33e3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.015653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.015559 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f197c-63bc-4bfc-b84a-6e620cea33e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.018048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.018022 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvbv\" (UniqueName: \"kubernetes.io/projected/997f197c-63bc-4bfc-b84a-6e620cea33e3-kube-api-access-xxvbv\") pod \"kuadrant-console-plugin-66c4fb575d-jr84g\" (UID: \"997f197c-63bc-4bfc-b84a-6e620cea33e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.087923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.087897 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" Oct 31 03:01:39.257495 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:01:39.257455 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:01:39.367464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:39.367354 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g"] Oct 31 03:01:39.370640 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:01:39.370607 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod997f197c_63bc_4bfc_b84a_6e620cea33e3.slice/crio-e0070793a1dcc31cd5b9e53d3bc2fd8d9b82f540229cf16c7d0b2e79f2a387da WatchSource:0}: Error finding container e0070793a1dcc31cd5b9e53d3bc2fd8d9b82f540229cf16c7d0b2e79f2a387da: Status 404 returned error can't find the container with id e0070793a1dcc31cd5b9e53d3bc2fd8d9b82f540229cf16c7d0b2e79f2a387da Oct 31 03:01:40.022278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:40.022223 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" event={"ID":"997f197c-63bc-4bfc-b84a-6e620cea33e3","Type":"ContainerStarted","Data":"e0070793a1dcc31cd5b9e53d3bc2fd8d9b82f540229cf16c7d0b2e79f2a387da"} Oct 31 03:01:41.030591 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:41.030551 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" event={"ID":"763496b7-9cc7-475a-96c9-86a30d8b68c5","Type":"ContainerStarted","Data":"e5a74071c03b6c60e3b8d14a37d72198309bc106ff8a50f1d2a68fa7141524fb"} Oct 31 03:01:41.031004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:41.030674 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:41.055282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:41.054916 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" podStartSLOduration=2.022434712 podStartE2EDuration="5.054899211s" totalImagesPullingTime="3.032464499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:41.053735718 +0000 UTC m=+384.750903022" watchObservedRunningTime="2026-10-31 03:01:41.054899211 +0000 UTC m=+384.752066532" Oct 31 03:01:46.052512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:46.052473 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" event={"ID":"997f197c-63bc-4bfc-b84a-6e620cea33e3","Type":"ContainerStarted","Data":"f1c89777db440c9fa41d9dbdfc72ac532232163655384d731ff3f8ee6f46ab9c"} Oct 31 03:01:46.106371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:46.106326 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jr84g" podStartSLOduration=1.880919483 podStartE2EDuration="8.106314278s" totalImagesPullingTime="6.225394795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:46.106012357 +0000 UTC m=+389.803179663" watchObservedRunningTime="2026-10-31 03:01:46.106314278 +0000 UTC m=+389.803481582" Oct 31 03:01:51.955668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.955627 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw"] Oct 31 03:01:51.960018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.959998 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:51.964313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.964287 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-nn4wq\"" Oct 31 03:01:51.964479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.964447 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 03:01:51.964540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.964466 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 03:01:51.965038 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.965021 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 03:01:51.965236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.965220 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:01:51.965341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.965331 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 03:01:51.971932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:51.971914 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw"] Oct 31 03:01:52.036388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.036362 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rlp29" Oct 31 03:01:52.122452 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.122427 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af3582f8-22df-4002-9035-92e681e15887-cert\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.122592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.122534 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfff\" (UniqueName: \"kubernetes.io/projected/af3582f8-22df-4002-9035-92e681e15887-kube-api-access-gpfff\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.122667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.122641 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af3582f8-22df-4002-9035-92e681e15887-manager-config\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.122731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.122720 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3582f8-22df-4002-9035-92e681e15887-metrics-certs\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.224890 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.224021 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfff\" (UniqueName: \"kubernetes.io/projected/af3582f8-22df-4002-9035-92e681e15887-kube-api-access-gpfff\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.224890 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.224163 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af3582f8-22df-4002-9035-92e681e15887-manager-config\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.224890 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.224243 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3582f8-22df-4002-9035-92e681e15887-metrics-certs\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.224890 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.224303 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af3582f8-22df-4002-9035-92e681e15887-cert\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.225279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.225163 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af3582f8-22df-4002-9035-92e681e15887-manager-config\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.228445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.228418 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af3582f8-22df-4002-9035-92e681e15887-cert\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.228567 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.228485 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3582f8-22df-4002-9035-92e681e15887-metrics-certs\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.236347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.236326 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfff\" (UniqueName: \"kubernetes.io/projected/af3582f8-22df-4002-9035-92e681e15887-kube-api-access-gpfff\") pod \"jobset-controller-manager-5d5bfcd6b9-9sclw\" (UID: \"af3582f8-22df-4002-9035-92e681e15887\") " pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.272306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.272269 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:52.546310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:52.546281 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw"] Oct 31 03:01:52.549275 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:01:52.549221 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3582f8_22df_4002_9035_92e681e15887.slice/crio-9b8783c1b86d0e1f6f3361822bd93de3cb527567b47380ba843ff9810f730145 WatchSource:0}: Error finding container 9b8783c1b86d0e1f6f3361822bd93de3cb527567b47380ba843ff9810f730145: Status 404 returned error can't find the container with id 9b8783c1b86d0e1f6f3361822bd93de3cb527567b47380ba843ff9810f730145 Oct 31 03:01:53.082339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:53.082301 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" event={"ID":"af3582f8-22df-4002-9035-92e681e15887","Type":"ContainerStarted","Data":"9b8783c1b86d0e1f6f3361822bd93de3cb527567b47380ba843ff9810f730145"} Oct 31 03:01:56.096398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:56.096321 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" event={"ID":"af3582f8-22df-4002-9035-92e681e15887","Type":"ContainerStarted","Data":"0cda5c4db1ae0de39dc0b4f1fa5a4f1204e04978f5c9288b827b2dde5903cafd"} Oct 31 03:01:56.096753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:56.096448 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:01:56.114344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:01:56.114282 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" podStartSLOduration=1.829651833 podStartE2EDuration="5.114264204s" totalImagesPullingTime="3.284612371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:56.113207791 +0000 UTC m=+399.810375095" watchObservedRunningTime="2026-10-31 03:01:56.114264204 +0000 UTC m=+399.811431507" Oct 31 03:02:07.105844 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:07.105803 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d5bfcd6b9-9sclw" Oct 31 03:02:09.250055 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:02:09.250017 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:02:30.199114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.199085 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7bvdz"] Oct 31 03:02:30.202424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.202404 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.225461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.225406 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 03:02:30.225837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.225797 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cb6pb\"" Oct 31 03:02:30.225989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.225969 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7bvdz"] Oct 31 03:02:30.240275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.240229 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-7tcqw"] Oct 31 03:02:30.243562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.243537 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.252767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.252741 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78"] Oct 31 03:02:30.256034 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.255866 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 03:02:30.256319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.256297 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.256405 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.256375 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mgwwr\"" Oct 31 03:02:30.257535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.257038 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 03:02:30.262744 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.262723 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-7tcqw"] Oct 31 03:02:30.270660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.270642 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:02:30.270743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.270669 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:02:30.276208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.276186 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78"] Oct 31 03:02:30.317755 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317729 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59247df-efcb-4217-8b5a-66282ba7d89f-metrics-certs\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.317889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317760 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.317889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317780 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjpx\" (UniqueName: \"kubernetes.io/projected/a5461b19-2316-454f-a955-73429d2e50a2-kube-api-access-xnjpx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.317889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317796 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d599f3d-829f-4268-beb8-739eb59882ec-tls-certs\") pod \"model-serving-api-75bd89fdfc-7bvdz\" (UID: \"7d599f3d-829f-4268-beb8-739eb59882ec\") " pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.318022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317954 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5461b19-2316-454f-a955-73429d2e50a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.318022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317982 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5461b19-2316-454f-a955-73429d2e50a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.318022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.317997 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.318022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.318013 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6lp\" (UniqueName: \"kubernetes.io/projected/7d599f3d-829f-4268-beb8-739eb59882ec-kube-api-access-7t6lp\") pod \"model-serving-api-75bd89fdfc-7bvdz\" (UID: \"7d599f3d-829f-4268-beb8-739eb59882ec\") " pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.318156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.318056 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59247df-efcb-4217-8b5a-66282ba7d89f-cert\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.318156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.318144 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.318221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.318167 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5461b19-2316-454f-a955-73429d2e50a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.318221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.318193 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56s2\" (UniqueName: \"kubernetes.io/projected/f59247df-efcb-4217-8b5a-66282ba7d89f-kube-api-access-f56s2\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.419466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419440 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5461b19-2316-454f-a955-73429d2e50a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.419574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419472 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56s2\" (UniqueName: \"kubernetes.io/projected/f59247df-efcb-4217-8b5a-66282ba7d89f-kube-api-access-f56s2\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.419574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419513 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59247df-efcb-4217-8b5a-66282ba7d89f-metrics-certs\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.419574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419534 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.419574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419551 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjpx\" (UniqueName: \"kubernetes.io/projected/a5461b19-2316-454f-a955-73429d2e50a2-kube-api-access-xnjpx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.419574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419568 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d599f3d-829f-4268-beb8-739eb59882ec-tls-certs\") pod \"model-serving-api-75bd89fdfc-7bvdz\" (UID: \"7d599f3d-829f-4268-beb8-739eb59882ec\") " pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.419798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419588 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5461b19-2316-454f-a955-73429d2e50a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.419798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419604 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5461b19-2316-454f-a955-73429d2e50a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.419798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419622 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.419798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419646 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6lp\" (UniqueName: \"kubernetes.io/projected/7d599f3d-829f-4268-beb8-739eb59882ec-kube-api-access-7t6lp\") pod \"model-serving-api-75bd89fdfc-7bvdz\" (UID: \"7d599f3d-829f-4268-beb8-739eb59882ec\") " pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.419994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419820 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59247df-efcb-4217-8b5a-66282ba7d89f-cert\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.419994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.419871 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.421004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.420979 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a5461b19-2316-454f-a955-73429d2e50a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.422777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.422754 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d599f3d-829f-4268-beb8-739eb59882ec-tls-certs\") pod \"model-serving-api-75bd89fdfc-7bvdz\" (UID: \"7d599f3d-829f-4268-beb8-739eb59882ec\") " pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.423225 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.423200 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.423337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.423223 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.423383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.423331 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a5461b19-2316-454f-a955-73429d2e50a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.423431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.423415 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5461b19-2316-454f-a955-73429d2e50a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.423550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.423531 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f59247df-efcb-4217-8b5a-66282ba7d89f-metrics-certs\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.423583 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.423540 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59247df-efcb-4217-8b5a-66282ba7d89f-cert\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.430954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.430920 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjpx\" (UniqueName: \"kubernetes.io/projected/a5461b19-2316-454f-a955-73429d2e50a2-kube-api-access-xnjpx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.430954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.430948 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5461b19-2316-454f-a955-73429d2e50a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dbq78\" (UID: \"a5461b19-2316-454f-a955-73429d2e50a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.431082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.430972 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6lp\" (UniqueName: \"kubernetes.io/projected/7d599f3d-829f-4268-beb8-739eb59882ec-kube-api-access-7t6lp\") pod \"model-serving-api-75bd89fdfc-7bvdz\" (UID: \"7d599f3d-829f-4268-beb8-739eb59882ec\") " pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.431389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.431374 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56s2\" (UniqueName: \"kubernetes.io/projected/f59247df-efcb-4217-8b5a-66282ba7d89f-kube-api-access-f56s2\") pod \"odh-model-controller-769875db56-7tcqw\" (UID: \"f59247df-efcb-4217-8b5a-66282ba7d89f\") " pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.515801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.515777 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:30.557720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.557685 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:30.572621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.572591 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:30.791403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.791197 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7bvdz"] Oct 31 03:02:30.794472 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:02:30.794442 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d599f3d_829f_4268_beb8_739eb59882ec.slice/crio-c8acaaecfd21a9e4ca9a186a586f3234a5fb66d62dc5d2590068fc4212da6462 WatchSource:0}: Error finding container c8acaaecfd21a9e4ca9a186a586f3234a5fb66d62dc5d2590068fc4212da6462: Status 404 returned error can't find the container with id c8acaaecfd21a9e4ca9a186a586f3234a5fb66d62dc5d2590068fc4212da6462 Oct 31 03:02:30.846676 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.846642 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-7tcqw"] Oct 31 03:02:30.849329 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:02:30.849303 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59247df_efcb_4217_8b5a_66282ba7d89f.slice/crio-152d34a20a5c203964af06c259a0e1b21f86180dec810d3729df3f2631347647 WatchSource:0}: Error finding container 152d34a20a5c203964af06c259a0e1b21f86180dec810d3729df3f2631347647: Status 404 returned error can't find the container with id 152d34a20a5c203964af06c259a0e1b21f86180dec810d3729df3f2631347647 Oct 31 03:02:30.863763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.863743 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78"] Oct 31 03:02:30.865928 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:02:30.865901 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5461b19_2316_454f_a955_73429d2e50a2.slice/crio-3d650cb25cc2617d385ad89a4273e99730d8c0b5dd4739d882f8b029cd1f565b WatchSource:0}: Error finding container 3d650cb25cc2617d385ad89a4273e99730d8c0b5dd4739d882f8b029cd1f565b: Status 404 returned error can't find the container with id 3d650cb25cc2617d385ad89a4273e99730d8c0b5dd4739d882f8b029cd1f565b Oct 31 03:02:30.867852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.867820 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:02:30.867932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:30.867881 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:02:31.249499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.249459 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" event={"ID":"7d599f3d-829f-4268-beb8-739eb59882ec","Type":"ContainerStarted","Data":"c8acaaecfd21a9e4ca9a186a586f3234a5fb66d62dc5d2590068fc4212da6462"} Oct 31 03:02:31.250983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.250953 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-7tcqw" event={"ID":"f59247df-efcb-4217-8b5a-66282ba7d89f","Type":"ContainerStarted","Data":"152d34a20a5c203964af06c259a0e1b21f86180dec810d3729df3f2631347647"} Oct 31 03:02:31.252741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.252716 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" event={"ID":"a5461b19-2316-454f-a955-73429d2e50a2","Type":"ContainerStarted","Data":"6b8163a4d6947fa56ec0463d38e64078e32e69d8b20ab80d59a60597cd054f8c"} Oct 31 03:02:31.252870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.252756 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" event={"ID":"a5461b19-2316-454f-a955-73429d2e50a2","Type":"ContainerStarted","Data":"3d650cb25cc2617d385ad89a4273e99730d8c0b5dd4739d882f8b029cd1f565b"} Oct 31 03:02:31.252931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.252895 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:31.254688 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.254668 4854 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dbq78 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:02:31.254813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.254716 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" podUID="a5461b19-2316-454f-a955-73429d2e50a2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:02:31.275212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:31.275163 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" podStartSLOduration=1.275146402 podStartE2EDuration="1.275146402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:31.274233816 +0000 UTC m=+434.971401158" watchObservedRunningTime="2026-10-31 03:02:31.275146402 +0000 UTC m=+434.972313709" Oct 31 03:02:32.260848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.260816 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dbq78" Oct 31 03:02:32.321850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.321803 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk"] Oct 31 03:02:32.322022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.321988 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" podUID="e5fa8120-4f33-4995-9b5e-adcfa96b3c16" containerName="discovery" containerID="cri-o://5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2" gracePeriod=30 Oct 31 03:02:32.610473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.610442 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:02:32.757630 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757600 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.757807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757654 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.757807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757678 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.757807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757707 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.757807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757737 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.757807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757757 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.757807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.757785 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token\") pod \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\" (UID: \"e5fa8120-4f33-4995-9b5e-adcfa96b3c16\") " Oct 31 03:02:32.759585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.759547 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:02:32.764031 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.764002 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:02:32.764141 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.764049 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:02:32.764235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.764209 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "kube-api-access-nj2hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:02:32.764361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.764333 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:02:32.764491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.764467 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:02:32.764651 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.764629 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig" pod "e5fa8120-4f33-4995-9b5e-adcfa96b3c16" (UID: "e5fa8120-4f33-4995-9b5e-adcfa96b3c16"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:02:32.859239 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859169 4854 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-ca-configmap\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.859239 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859204 4854 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-kubeconfig\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.859239 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859220 4854 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-csr-dns-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.859239 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859237 4854 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-cacerts\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.859575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859277 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj2hf\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-kube-api-access-nj2hf\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.859575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859291 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:32.859575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:32.859301 4854 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fa8120-4f33-4995-9b5e-adcfa96b3c16-local-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:33.265993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.265949 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" event={"ID":"7d599f3d-829f-4268-beb8-739eb59882ec","Type":"ContainerStarted","Data":"7dd64a500eaa665889d9ae6a7ae54d9fabf271e31e82dbd64efec4fa2d5c33c8"} Oct 31 03:02:33.266462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.266021 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:33.267423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.267394 4854 generic.go:356] "Generic (PLEG): container finished" podID="e5fa8120-4f33-4995-9b5e-adcfa96b3c16" containerID="5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2" exitCode=0 Oct 31 03:02:33.267550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.267430 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" Oct 31 03:02:33.267550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.267464 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" event={"ID":"e5fa8120-4f33-4995-9b5e-adcfa96b3c16","Type":"ContainerDied","Data":"5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2"} Oct 31 03:02:33.267550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.267501 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk" event={"ID":"e5fa8120-4f33-4995-9b5e-adcfa96b3c16","Type":"ContainerDied","Data":"38867c5c7b566c4740e8572d84fce2eae9ff18524a14779f8635d0871d24b187"} Oct 31 03:02:33.267550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.267520 4854 scope.go:122] "RemoveContainer" containerID="5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2" Oct 31 03:02:33.276776 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.276750 4854 scope.go:122] "RemoveContainer" containerID="5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2" Oct 31 03:02:33.277050 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:02:33.277030 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2\": container with ID starting with 5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2 not found: ID does not exist" containerID="5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2" Oct 31 03:02:33.277119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.277058 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2"} err="failed to get container status \"5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2\": rpc error: code = NotFound desc = could not find container \"5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2\": container with ID starting with 5246332d15531877003d2bf9a1107a64839d1327398ce5215face947dbb7c7c2 not found: ID does not exist" Oct 31 03:02:33.283708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.283665 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" podStartSLOduration=1.040556017 podStartE2EDuration="3.283652021s" totalImagesPullingTime="2.243096004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:33.282983349 +0000 UTC m=+436.980150654" watchObservedRunningTime="2026-10-31 03:02:33.283652021 +0000 UTC m=+436.980819326" Oct 31 03:02:33.295004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.294980 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk"] Oct 31 03:02:33.299043 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:33.299023 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dxkkk"] Oct 31 03:02:34.273442 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:34.273405 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-7tcqw" event={"ID":"f59247df-efcb-4217-8b5a-66282ba7d89f","Type":"ContainerStarted","Data":"9ca1250ee57dfc77d9bf13c95be622f1dc97d894ca1e8d2e72b89decf8235993"} Oct 31 03:02:34.273900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:34.273522 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:02:34.292433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:34.292379 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-7tcqw" podStartSLOduration=1.322334695 podStartE2EDuration="4.292364397s" totalImagesPullingTime="2.970029702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:34.291310027 +0000 UTC m=+437.988477333" watchObservedRunningTime="2026-10-31 03:02:34.292364397 +0000 UTC m=+437.989531702" Oct 31 03:02:34.454323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:34.454293 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fa8120-4f33-4995-9b5e-adcfa96b3c16" path="/var/lib/kubelet/pods/e5fa8120-4f33-4995-9b5e-adcfa96b3c16/volumes" Oct 31 03:02:39.251715 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:02:39.251678 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:02:44.278871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:44.278837 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7bvdz" Oct 31 03:02:45.281635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:02:45.281603 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-7tcqw" Oct 31 03:03:09.252444 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:03:09.252395 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:03:39.252063 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:03:39.251973 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:03:41.695776 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.695747 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz"] Oct 31 03:03:41.696183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.696064 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e5fa8120-4f33-4995-9b5e-adcfa96b3c16" containerName="discovery" Oct 31 03:03:41.696183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.696074 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fa8120-4f33-4995-9b5e-adcfa96b3c16" containerName="discovery" Oct 31 03:03:41.696183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.696133 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fa8120-4f33-4995-9b5e-adcfa96b3c16" containerName="discovery" Oct 31 03:03:41.699133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.699117 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.709751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.709729 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz"] Oct 31 03:03:41.718708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.718684 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:03:41.718708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.718695 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 03:03:41.718858 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.718763 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:03:41.739687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.739665 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:03:41.739827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.739813 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:03:41.778891 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.778852 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.778981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.778964 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8zd\" (UniqueName: \"kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.779019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.778990 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.779066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.779051 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.880178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.880144 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8zd\" (UniqueName: \"kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.880293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.880184 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.880293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.880214 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.880293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.880266 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.880893 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.880875 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.882780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.882764 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.882940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.882921 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:41.894426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:41.894402 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8zd\" (UniqueName: \"kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd\") pod \"maas-api-5ccbbc67c7-hmtnz\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:42.014642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:42.014615 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:42.287151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:42.287078 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz"] Oct 31 03:03:42.290159 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:03:42.290135 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6d1a73_179d_4794_bd39_169c04ced3f7.slice/crio-587c906a08749914db1e99dad240def392cbf38142a4c976aadbed696818dfbc WatchSource:0}: Error finding container 587c906a08749914db1e99dad240def392cbf38142a4c976aadbed696818dfbc: Status 404 returned error can't find the container with id 587c906a08749914db1e99dad240def392cbf38142a4c976aadbed696818dfbc Oct 31 03:03:42.557590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:42.557511 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" event={"ID":"7f6d1a73-179d-4794-bd39-169c04ced3f7","Type":"ContainerStarted","Data":"587c906a08749914db1e99dad240def392cbf38142a4c976aadbed696818dfbc"} Oct 31 03:03:45.583092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:45.583058 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" event={"ID":"7f6d1a73-179d-4794-bd39-169c04ced3f7","Type":"ContainerStarted","Data":"bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26"} Oct 31 03:03:45.583599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:45.583206 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:03:45.603057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:45.603011 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" podStartSLOduration=2.116018234 podStartE2EDuration="4.602995617s" totalImagesPullingTime="2.486977383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:45.60216644 +0000 UTC m=+509.299333746" watchObservedRunningTime="2026-10-31 03:03:45.602995617 +0000 UTC m=+509.300162922" Oct 31 03:03:51.592566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:03:51.592538 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:04:09.251062 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:04:09.251026 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:04:39.251893 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:04:39.251859 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:04:40.991844 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:40.991809 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5"] Oct 31 03:04:40.995190 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:40.995171 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:40.999566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:40.999542 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 03:04:40.999869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:40.999854 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 03:04:41.001810 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.001783 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-88f92\"" Oct 31 03:04:41.002046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.002025 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 03:04:41.007358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.007337 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5"] Oct 31 03:04:41.032083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.032056 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.032188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.032097 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.032188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.032124 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.032188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.032168 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.032419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.032308 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f7442a-d473-49cc-aa44-0cb836085273-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.032419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.032342 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kt5l\" (UniqueName: \"kubernetes.io/projected/94f7442a-d473-49cc-aa44-0cb836085273-kube-api-access-2kt5l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.132920 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.132891 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.132925 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.132951 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f7442a-d473-49cc-aa44-0cb836085273-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.132967 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kt5l\" (UniqueName: \"kubernetes.io/projected/94f7442a-d473-49cc-aa44-0cb836085273-kube-api-access-2kt5l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.133002 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.133033 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.133346 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.133391 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.133550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.133531 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.135422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.135402 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f7442a-d473-49cc-aa44-0cb836085273-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.135743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.135728 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f7442a-d473-49cc-aa44-0cb836085273-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.144536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.144513 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kt5l\" (UniqueName: \"kubernetes.io/projected/94f7442a-d473-49cc-aa44-0cb836085273-kube-api-access-2kt5l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5\" (UID: \"94f7442a-d473-49cc-aa44-0cb836085273\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.311999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.311935 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:41.586385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.586311 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5"] Oct 31 03:04:41.589441 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:04:41.589399 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f7442a_d473_49cc_aa44_0cb836085273.slice/crio-19f256255a347275deb3d8f09abd82f2b24a112dc5fa00cf63f750fce13d28bf WatchSource:0}: Error finding container 19f256255a347275deb3d8f09abd82f2b24a112dc5fa00cf63f750fce13d28bf: Status 404 returned error can't find the container with id 19f256255a347275deb3d8f09abd82f2b24a112dc5fa00cf63f750fce13d28bf Oct 31 03:04:41.815771 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:41.815741 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" event={"ID":"94f7442a-d473-49cc-aa44-0cb836085273","Type":"ContainerStarted","Data":"19f256255a347275deb3d8f09abd82f2b24a112dc5fa00cf63f750fce13d28bf"} Oct 31 03:04:42.821584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:42.821552 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" event={"ID":"94f7442a-d473-49cc-aa44-0cb836085273","Type":"ContainerStarted","Data":"a40188d1f2ff6b3264a51bdc837eb1516e369b81370cce954b48d841c75e6f4f"} Oct 31 03:04:42.841650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:42.841602 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" podStartSLOduration=1.717204349 podStartE2EDuration="2.8415879s" totalImagesPullingTime="1.124383551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:42.840829518 +0000 UTC m=+566.537996822" watchObservedRunningTime="2026-10-31 03:04:42.8415879 +0000 UTC m=+566.538755206" Oct 31 03:04:43.312608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:43.312582 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:44.715858 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.715827 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5"] Oct 31 03:04:44.719571 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.719548 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.731617 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.731590 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 03:04:44.733071 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.733047 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5"] Oct 31 03:04:44.764995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.764966 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.765106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.764998 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.765106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.765033 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b332bc-d96f-4af7-871e-be4a9a439e9a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.765180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.765094 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrh5\" (UniqueName: \"kubernetes.io/projected/c2b332bc-d96f-4af7-871e-be4a9a439e9a-kube-api-access-fdrh5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.765180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.765158 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.765244 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.765190 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.865774 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.865741 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b332bc-d96f-4af7-871e-be4a9a439e9a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.865936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.865781 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrh5\" (UniqueName: \"kubernetes.io/projected/c2b332bc-d96f-4af7-871e-be4a9a439e9a-kube-api-access-fdrh5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.865936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.865839 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.865936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.865875 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.865936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.865911 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.866121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.865938 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.866392 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.866366 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.866545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.866416 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.866545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.866477 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.868268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.868220 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b332bc-d96f-4af7-871e-be4a9a439e9a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.868387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.868370 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b332bc-d96f-4af7-871e-be4a9a439e9a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:44.879815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:44.879795 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrh5\" (UniqueName: \"kubernetes.io/projected/c2b332bc-d96f-4af7-871e-be4a9a439e9a-kube-api-access-fdrh5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wmfx5\" (UID: \"c2b332bc-d96f-4af7-871e-be4a9a439e9a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:45.033693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:45.033659 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:45.308102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:45.308079 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5"] Oct 31 03:04:45.310769 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:04:45.310745 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b332bc_d96f_4af7_871e_be4a9a439e9a.slice/crio-c8331769dbbbcae211bc75819f859daaf168a17389f0226586f8a1d035e73c9e WatchSource:0}: Error finding container c8331769dbbbcae211bc75819f859daaf168a17389f0226586f8a1d035e73c9e: Status 404 returned error can't find the container with id c8331769dbbbcae211bc75819f859daaf168a17389f0226586f8a1d035e73c9e Oct 31 03:04:45.836390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:45.836356 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" event={"ID":"c2b332bc-d96f-4af7-871e-be4a9a439e9a","Type":"ContainerStarted","Data":"0e6c0b7ceb4887ae293f7e9d60635ba155374ecf65eef69302c3793ced0a0e0a"} Oct 31 03:04:45.836748 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:45.836405 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" event={"ID":"c2b332bc-d96f-4af7-871e-be4a9a439e9a","Type":"ContainerStarted","Data":"c8331769dbbbcae211bc75819f859daaf168a17389f0226586f8a1d035e73c9e"} Oct 31 03:04:45.856935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:45.856851 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" podStartSLOduration=1.576717956 podStartE2EDuration="1.856837373s" totalImagesPullingTime="280.119417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:45.855425202 +0000 UTC m=+569.552592530" watchObservedRunningTime="2026-10-31 03:04:45.856837373 +0000 UTC m=+569.554004677" Oct 31 03:04:46.034290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:46.034262 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:47.528881 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.528852 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jlgpj"] Oct 31 03:04:47.532240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.532221 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.537226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.537204 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 03:04:47.544369 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.544346 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jlgpj"] Oct 31 03:04:47.586756 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.586734 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.586861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.586776 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-home\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.586861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.586796 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.586861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.586852 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs25l\" (UniqueName: \"kubernetes.io/projected/9ca899b4-82ad-4c0f-9654-f9912c269a4e-kube-api-access-hs25l\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.586969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.586917 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.586969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.586938 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca899b4-82ad-4c0f-9654-f9912c269a4e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688225 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688199 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688234 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca899b4-82ad-4c0f-9654-f9912c269a4e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688310 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688380 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-home\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688405 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688441 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs25l\" (UniqueName: \"kubernetes.io/projected/9ca899b4-82ad-4c0f-9654-f9912c269a4e-kube-api-access-hs25l\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688624 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688756 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688735 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-home\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.688808 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.688770 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.690668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.690648 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca899b4-82ad-4c0f-9654-f9912c269a4e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.690800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.690783 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca899b4-82ad-4c0f-9654-f9912c269a4e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.733425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.733404 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs25l\" (UniqueName: \"kubernetes.io/projected/9ca899b4-82ad-4c0f-9654-f9912c269a4e-kube-api-access-hs25l\") pod \"echo-simulator-kserve-64b6f8f96c-jlgpj\" (UID: \"9ca899b4-82ad-4c0f-9654-f9912c269a4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:47.844839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:47.844775 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:48.121041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.120970 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jlgpj"] Oct 31 03:04:48.124021 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:04:48.123991 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca899b4_82ad_4c0f_9654_f9912c269a4e.slice/crio-62ef9ca304eddaef1d278a86475cbff5c759ca77436a3624a4bfca33109eeadc WatchSource:0}: Error finding container 62ef9ca304eddaef1d278a86475cbff5c759ca77436a3624a4bfca33109eeadc: Status 404 returned error can't find the container with id 62ef9ca304eddaef1d278a86475cbff5c759ca77436a3624a4bfca33109eeadc Oct 31 03:04:48.860367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.858665 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt"] Oct 31 03:04:48.864660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.864629 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" event={"ID":"9ca899b4-82ad-4c0f-9654-f9912c269a4e","Type":"ContainerStarted","Data":"62ef9ca304eddaef1d278a86475cbff5c759ca77436a3624a4bfca33109eeadc"} Oct 31 03:04:48.864816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.864721 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:48.873276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.873045 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt"] Oct 31 03:04:48.873276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.873222 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 03:04:48.999198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.999167 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679a142-f04f-4b6f-b25e-ed237eb68f78-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:48.999435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.999277 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:48.999435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.999315 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr26t\" (UniqueName: \"kubernetes.io/projected/c679a142-f04f-4b6f-b25e-ed237eb68f78-kube-api-access-kr26t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:48.999435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.999350 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:48.999435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.999423 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:48.999617 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:48.999464 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.100681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.100643 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.100846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.100714 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.100846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.100752 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679a142-f04f-4b6f-b25e-ed237eb68f78-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.100846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.100824 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.101008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.100861 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr26t\" (UniqueName: \"kubernetes.io/projected/c679a142-f04f-4b6f-b25e-ed237eb68f78-kube-api-access-kr26t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.101008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.100899 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.101354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.101329 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.102919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.102242 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.102919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.102873 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.104003 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.103976 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c679a142-f04f-4b6f-b25e-ed237eb68f78-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.106081 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.106044 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c679a142-f04f-4b6f-b25e-ed237eb68f78-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.114541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.114496 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr26t\" (UniqueName: \"kubernetes.io/projected/c679a142-f04f-4b6f-b25e-ed237eb68f78-kube-api-access-kr26t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt\" (UID: \"c679a142-f04f-4b6f-b25e-ed237eb68f78\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.181511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.181465 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:49.465903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.465840 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt"] Oct 31 03:04:49.469312 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:04:49.469282 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc679a142_f04f_4b6f_b25e_ed237eb68f78.slice/crio-d64ad934e38419f2a212d107567b49a8478ebc5e789ff10c580715c15c461c6f WatchSource:0}: Error finding container d64ad934e38419f2a212d107567b49a8478ebc5e789ff10c580715c15c461c6f: Status 404 returned error can't find the container with id d64ad934e38419f2a212d107567b49a8478ebc5e789ff10c580715c15c461c6f Oct 31 03:04:49.865216 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.865182 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" event={"ID":"c679a142-f04f-4b6f-b25e-ed237eb68f78","Type":"ContainerStarted","Data":"ca8378d73649ec2d01216126379adc0cb7d1bd8182d5b353244633978b73613a"} Oct 31 03:04:49.865542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.865236 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" event={"ID":"c679a142-f04f-4b6f-b25e-ed237eb68f78","Type":"ContainerStarted","Data":"d64ad934e38419f2a212d107567b49a8478ebc5e789ff10c580715c15c461c6f"} Oct 31 03:04:49.885575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:49.885533 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" podStartSLOduration=1.622586495 podStartE2EDuration="1.885519295s" totalImagesPullingTime="262.9328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:49.884224906 +0000 UTC m=+573.581392212" watchObservedRunningTime="2026-10-31 03:04:49.885519295 +0000 UTC m=+573.582686630" Oct 31 03:04:50.182422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:50.182391 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:50.871053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:50.871018 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" event={"ID":"9ca899b4-82ad-4c0f-9654-f9912c269a4e","Type":"ContainerStarted","Data":"64ea984aa86f1f739db25d0c310b96be3ac84ce06336714ad10b1ed67e2a488b"} Oct 31 03:04:50.892550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:50.892499 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" podStartSLOduration=2.195389676 podStartE2EDuration="3.892486473s" totalImagesPullingTime="1.697096797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:50.890685467 +0000 UTC m=+574.587852771" watchObservedRunningTime="2026-10-31 03:04:50.892486473 +0000 UTC m=+574.589653814" Oct 31 03:04:51.312369 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:51.312339 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:51.324904 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:51.324877 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:51.845431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:51.845399 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:51.886706 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:51.886683 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5" Oct 31 03:04:55.034379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:55.034345 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:55.046509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:55.046489 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:55.906177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:55.906143 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" Oct 31 03:04:57.845852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:57.845760 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:57.858625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:57.858599 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:57.914638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:57.914611 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" Oct 31 03:04:59.182053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:59.182016 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:59.194675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:59.194654 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:04:59.923565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:04:59.923532 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" Oct 31 03:05:01.671796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.671770 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2"] Oct 31 03:05:01.675468 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.675446 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.690892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.690864 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2"] Oct 31 03:05:01.766910 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.766879 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zfwtw\"" Oct 31 03:05:01.799950 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.799914 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.799965 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800038 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800083 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg9v\" (UniqueName: \"kubernetes.io/projected/b46a699c-a9b7-4a4a-a8b3-43d37829903a-kube-api-access-5gg9v\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800162 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800207 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800231 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800321 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.800357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.800351 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901079 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901114 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901137 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901179 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901197 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901226 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg9v\" (UniqueName: \"kubernetes.io/projected/b46a699c-a9b7-4a4a-a8b3-43d37829903a-kube-api-access-5gg9v\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901276 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901319 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901354 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901539 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901568 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.901937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901910 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.902061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901943 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.902061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.901967 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.903905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.903872 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.903996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.903980 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.912894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.912863 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46a699c-a9b7-4a4a-a8b3-43d37829903a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.913534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.913510 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg9v\" (UniqueName: \"kubernetes.io/projected/b46a699c-a9b7-4a4a-a8b3-43d37829903a-kube-api-access-5gg9v\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mcqz2\" (UID: \"b46a699c-a9b7-4a4a-a8b3-43d37829903a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:01.993487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:01.993462 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:02.276041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:02.276018 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2"] Oct 31 03:05:02.278761 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:05:02.278731 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46a699c_a9b7_4a4a_a8b3_43d37829903a.slice/crio-b9efd6b5a8cd57cdf9944b3f4326dc4cd5896e57c654d3ef78d05d3bcaa42b9a WatchSource:0}: Error finding container b9efd6b5a8cd57cdf9944b3f4326dc4cd5896e57c654d3ef78d05d3bcaa42b9a: Status 404 returned error can't find the container with id b9efd6b5a8cd57cdf9944b3f4326dc4cd5896e57c654d3ef78d05d3bcaa42b9a Oct 31 03:05:02.926832 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:02.926800 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" event={"ID":"b46a699c-a9b7-4a4a-a8b3-43d37829903a","Type":"ContainerStarted","Data":"b9efd6b5a8cd57cdf9944b3f4326dc4cd5896e57c654d3ef78d05d3bcaa42b9a"} Oct 31 03:05:05.770614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:05.770583 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:05:05.770906 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:05.770653 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:05:05.770906 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:05.770683 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:05:05.944230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:05.944191 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" event={"ID":"b46a699c-a9b7-4a4a-a8b3-43d37829903a","Type":"ContainerStarted","Data":"daf73a684d7e21c2ad8c46d2f6dda756f31d493ccbf94bd22f070a87c6916d48"} Oct 31 03:05:05.966569 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:05.966520 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podStartSLOduration=1.477467958 podStartE2EDuration="4.966506572s" totalImagesPullingTime="3.489038614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:05.965675341 +0000 UTC m=+589.662842644" watchObservedRunningTime="2026-10-31 03:05:05.966506572 +0000 UTC m=+589.663673876" Oct 31 03:05:05.994488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:05.994457 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:07.015307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.015234 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:07.015714 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.015332 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:07.322443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.322373 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7"] Oct 31 03:05:07.326560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.326544 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.330799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.330759 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 03:05:07.330799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.330763 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 03:05:07.330799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.330794 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-qk4z2\"" Oct 31 03:05:07.339729 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.339704 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7"] Oct 31 03:05:07.450907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.450872 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-z2zz7\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.450907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.450907 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xxk\" (UniqueName: \"kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk\") pod \"ai-gateway-controller-6cf7bcbfb6-z2zz7\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.552318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.552285 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-z2zz7\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.552465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.552325 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77xxk\" (UniqueName: \"kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk\") pod \"ai-gateway-controller-6cf7bcbfb6-z2zz7\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.555302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.555279 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-z2zz7\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.566949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.566926 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xxk\" (UniqueName: \"kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk\") pod \"ai-gateway-controller-6cf7bcbfb6-z2zz7\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.640914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.640858 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:07.935837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.935762 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7"] Oct 31 03:05:07.939467 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:05:07.939438 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c913834_e613_498a_9f8b_e72a3778bc7d.slice/crio-b609824423d6a26f95f08700acf0b8f0629a6a9a2214690fac8fa432f246709c WatchSource:0}: Error finding container b609824423d6a26f95f08700acf0b8f0629a6a9a2214690fac8fa432f246709c: Status 404 returned error can't find the container with id b609824423d6a26f95f08700acf0b8f0629a6a9a2214690fac8fa432f246709c Oct 31 03:05:07.954191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:07.954161 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" event={"ID":"2c913834-e613-498a-9f8b-e72a3778bc7d","Type":"ContainerStarted","Data":"b609824423d6a26f95f08700acf0b8f0629a6a9a2214690fac8fa432f246709c"} Oct 31 03:05:08.014274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:08.004638 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:08.014274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:08.004705 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:09.014392 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:09.014354 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:09.014881 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:09.014421 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:09.249659 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:05:09.249617 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:05:10.010410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:10.010379 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:10.010636 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:10.010608 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:10.952993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:10.952962 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7hkl8"] Oct 31 03:05:10.956623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:10.956606 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:10.968894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:10.968875 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7hkl8"] Oct 31 03:05:10.971063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:10.971039 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" event={"ID":"2c913834-e613-498a-9f8b-e72a3778bc7d","Type":"ContainerStarted","Data":"f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67"} Oct 31 03:05:11.022144 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.022118 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 03:05:11.023147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.023123 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6nw55\"" Oct 31 03:05:11.023633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.023610 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:11.023791 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.023769 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:11.038197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.038152 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" podStartSLOduration=2.041368321 podStartE2EDuration="4.038136143s" totalImagesPullingTime="1.996767822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:11.037036155 +0000 UTC m=+594.734203460" watchObservedRunningTime="2026-10-31 03:05:11.038136143 +0000 UTC m=+594.735303447" Oct 31 03:05:11.084481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.084455 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a631e7-c686-4a5a-98ef-e963e34cd85a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7hkl8\" (UID: \"04a631e7-c686-4a5a-98ef-e963e34cd85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.084717 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.084696 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfn5x\" (UniqueName: \"kubernetes.io/projected/04a631e7-c686-4a5a-98ef-e963e34cd85a-kube-api-access-jfn5x\") pod \"payload-processing-55fbbbf9dc-7hkl8\" (UID: \"04a631e7-c686-4a5a-98ef-e963e34cd85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.186224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.186186 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a631e7-c686-4a5a-98ef-e963e34cd85a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7hkl8\" (UID: \"04a631e7-c686-4a5a-98ef-e963e34cd85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.186424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.186270 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfn5x\" (UniqueName: \"kubernetes.io/projected/04a631e7-c686-4a5a-98ef-e963e34cd85a-kube-api-access-jfn5x\") pod \"payload-processing-55fbbbf9dc-7hkl8\" (UID: \"04a631e7-c686-4a5a-98ef-e963e34cd85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.186884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.186859 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a631e7-c686-4a5a-98ef-e963e34cd85a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7hkl8\" (UID: \"04a631e7-c686-4a5a-98ef-e963e34cd85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.215316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.215201 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfn5x\" (UniqueName: \"kubernetes.io/projected/04a631e7-c686-4a5a-98ef-e963e34cd85a-kube-api-access-jfn5x\") pod \"payload-processing-55fbbbf9dc-7hkl8\" (UID: \"04a631e7-c686-4a5a-98ef-e963e34cd85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.270001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.269973 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:11.566397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.566368 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7hkl8"] Oct 31 03:05:11.569710 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:05:11.569685 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a631e7_c686_4a5a_98ef_e963e34cd85a.slice/crio-0898c9b0c7785f6935fe67c59003f9b82bfa48596dc29cb52a4b76b9cba517a4 WatchSource:0}: Error finding container 0898c9b0c7785f6935fe67c59003f9b82bfa48596dc29cb52a4b76b9cba517a4: Status 404 returned error can't find the container with id 0898c9b0c7785f6935fe67c59003f9b82bfa48596dc29cb52a4b76b9cba517a4 Oct 31 03:05:11.976469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.976390 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" event={"ID":"04a631e7-c686-4a5a-98ef-e963e34cd85a","Type":"ContainerStarted","Data":"0898c9b0c7785f6935fe67c59003f9b82bfa48596dc29cb52a4b76b9cba517a4"} Oct 31 03:05:11.976841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.976531 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:11.993961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.993933 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:11.994069 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:11.993979 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:12.994354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:12.994325 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:05:12.994712 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:12.994388 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Oct 31 03:05:13.988047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:13.988011 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" event={"ID":"04a631e7-c686-4a5a-98ef-e963e34cd85a","Type":"ContainerStarted","Data":"f9b5799270560be5c2e85ace820a4a3f23259ddecd61ec4b25f59016e01d389a"} Oct 31 03:05:13.988211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:13.988084 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:14.007499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:14.007383 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" podStartSLOduration=1.7039732349999999 podStartE2EDuration="4.007368188s" totalImagesPullingTime="2.303394953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:14.00607464 +0000 UTC m=+597.703241944" watchObservedRunningTime="2026-10-31 03:05:14.007368188 +0000 UTC m=+597.704535492" Oct 31 03:05:14.994261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:14.994215 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:05:14.994467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:14.994283 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:05:16.035426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.035387 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:05:16.035824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.035470 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:05:16.252280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.252231 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:16.252645 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.252612 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:16.253373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.253353 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" Oct 31 03:05:16.487855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.487829 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:05:16.488027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.487943 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:05:16.684454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.684428 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577"] Oct 31 03:05:16.688243 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.688227 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.701942 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.701921 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577"] Oct 31 03:05:16.763648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763578 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.763648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763609 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.763648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763629 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.763864 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763775 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.763864 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763813 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.763940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763897 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.763940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763928 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.764015 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.763961 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj9q\" (UniqueName: \"kubernetes.io/projected/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-kube-api-access-vsj9q\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.764050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.764032 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864623 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864663 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864688 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864735 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864763 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864800 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864834 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.864888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864882 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj9q\" (UniqueName: \"kubernetes.io/projected/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-kube-api-access-vsj9q\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.865151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.864922 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.865151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.865045 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.865151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.865114 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.865337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.865195 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.865423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.865405 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.865550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.865534 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.867324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.867302 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.867479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.867464 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.876563 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.876543 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:16.876670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:16.876546 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj9q\" (UniqueName: \"kubernetes.io/projected/3621ef10-eb7d-4e57-9c29-5f3598cae2d9-kube-api-access-vsj9q\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kc577\" (UID: \"3621ef10-eb7d-4e57-9c29-5f3598cae2d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:17.001358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:17.001329 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:17.280171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:17.280138 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577"] Oct 31 03:05:17.282967 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:05:17.282943 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3621ef10_eb7d_4e57_9c29_5f3598cae2d9.slice/crio-0625b952e7ed365a2055e135e1be76a61be0b261d35372eb281c9f02796245a8 WatchSource:0}: Error finding container 0625b952e7ed365a2055e135e1be76a61be0b261d35372eb281c9f02796245a8: Status 404 returned error can't find the container with id 0625b952e7ed365a2055e135e1be76a61be0b261d35372eb281c9f02796245a8 Oct 31 03:05:17.285121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:17.285092 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:05:17.285226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:17.285149 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:05:17.285226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:17.285176 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:05:18.019149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:18.019105 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" event={"ID":"3621ef10-eb7d-4e57-9c29-5f3598cae2d9","Type":"ContainerStarted","Data":"1f56b3271ec413649a11ff197a9f39462cb9c891cccaded57814e018aa33fe3a"} Oct 31 03:05:18.019432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:18.019201 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" event={"ID":"3621ef10-eb7d-4e57-9c29-5f3598cae2d9","Type":"ContainerStarted","Data":"0625b952e7ed365a2055e135e1be76a61be0b261d35372eb281c9f02796245a8"} Oct 31 03:05:18.045588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:18.045541 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podStartSLOduration=2.045527275 podStartE2EDuration="2.045527275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:18.044385082 +0000 UTC m=+601.741552388" watchObservedRunningTime="2026-10-31 03:05:18.045527275 +0000 UTC m=+601.742694580" Oct 31 03:05:19.014343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:19.014308 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:19.014811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:19.014785 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:05:19.014868 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:19.014843 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:05:20.008767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:20.008731 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:05:20.009049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:20.009016 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:05:21.006664 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:21.006626 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:05:21.007265 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:21.007216 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:05:22.014587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:22.014558 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:05:22.014987 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:22.014616 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:05:22.984124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:22.984096 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:23.007640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.007607 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:05:23.007870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.007844 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:05:23.262821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.262748 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79bd86dfd9-966ss"] Oct 31 03:05:23.271933 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.271906 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.280566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.280450 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79bd86dfd9-966ss"] Oct 31 03:05:23.342110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.342081 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a92420e-917f-40b9-8b4c-2095dc028257-metrics-certs\") pod \"ai-gateway-controller-79bd86dfd9-966ss\" (UID: \"7a92420e-917f-40b9-8b4c-2095dc028257\") " pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.342234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.342158 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9vc\" (UniqueName: \"kubernetes.io/projected/7a92420e-917f-40b9-8b4c-2095dc028257-kube-api-access-nq9vc\") pod \"ai-gateway-controller-79bd86dfd9-966ss\" (UID: \"7a92420e-917f-40b9-8b4c-2095dc028257\") " pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.442891 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.442861 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a92420e-917f-40b9-8b4c-2095dc028257-metrics-certs\") pod \"ai-gateway-controller-79bd86dfd9-966ss\" (UID: \"7a92420e-917f-40b9-8b4c-2095dc028257\") " pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.443025 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.442952 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9vc\" (UniqueName: \"kubernetes.io/projected/7a92420e-917f-40b9-8b4c-2095dc028257-kube-api-access-nq9vc\") pod \"ai-gateway-controller-79bd86dfd9-966ss\" (UID: \"7a92420e-917f-40b9-8b4c-2095dc028257\") " pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.445420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.445396 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a92420e-917f-40b9-8b4c-2095dc028257-metrics-certs\") pod \"ai-gateway-controller-79bd86dfd9-966ss\" (UID: \"7a92420e-917f-40b9-8b4c-2095dc028257\") " pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.454836 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.454812 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9vc\" (UniqueName: \"kubernetes.io/projected/7a92420e-917f-40b9-8b4c-2095dc028257-kube-api-access-nq9vc\") pod \"ai-gateway-controller-79bd86dfd9-966ss\" (UID: \"7a92420e-917f-40b9-8b4c-2095dc028257\") " pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.588028 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.587972 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:23.862375 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:23.862351 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79bd86dfd9-966ss"] Oct 31 03:05:23.865876 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:05:23.865852 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a92420e_917f_40b9_8b4c_2095dc028257.slice/crio-2892ede07c7410e525de993a7441073bbd5aeaa57f503607828f28f2632978ac WatchSource:0}: Error finding container 2892ede07c7410e525de993a7441073bbd5aeaa57f503607828f28f2632978ac: Status 404 returned error can't find the container with id 2892ede07c7410e525de993a7441073bbd5aeaa57f503607828f28f2632978ac Oct 31 03:05:24.002595 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:24.002562 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 03:05:24.002734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:24.002610 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 03:05:24.048724 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:24.048700 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" event={"ID":"7a92420e-917f-40b9-8b4c-2095dc028257","Type":"ContainerStarted","Data":"2892ede07c7410e525de993a7441073bbd5aeaa57f503607828f28f2632978ac"} Oct 31 03:05:24.996491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:24.996460 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-7hkl8" Oct 31 03:05:25.056000 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:25.055967 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" event={"ID":"7a92420e-917f-40b9-8b4c-2095dc028257","Type":"ContainerStarted","Data":"89b73865028df1e841f01a81b1cd780c0fb2749d789384101625bc6f49a7df2b"} Oct 31 03:05:25.056189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:25.056064 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:25.074407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:25.074363 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" podStartSLOduration=1.64213097 podStartE2EDuration="2.074348883s" totalImagesPullingTime="432.217913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:25.072596737 +0000 UTC m=+608.769764042" watchObservedRunningTime="2026-10-31 03:05:25.074348883 +0000 UTC m=+608.771516187" Oct 31 03:05:25.437860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:25.437766 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:25.438174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:25.438136 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:25.438904 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:25.438883 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" Oct 31 03:05:36.063179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.063148 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79bd86dfd9-966ss" Oct 31 03:05:36.111760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.111730 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7"] Oct 31 03:05:36.111958 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.111921 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" podUID="2c913834-e613-498a-9f8b-e72a3778bc7d" containerName="manager" containerID="cri-o://f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67" gracePeriod=30 Oct 31 03:05:36.356973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.356946 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:36.458240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.458210 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk\" (UniqueName: \"kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk\") pod \"2c913834-e613-498a-9f8b-e72a3778bc7d\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " Oct 31 03:05:36.458420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.458303 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs\") pod \"2c913834-e613-498a-9f8b-e72a3778bc7d\" (UID: \"2c913834-e613-498a-9f8b-e72a3778bc7d\") " Oct 31 03:05:36.460558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.460529 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs" pod "2c913834-e613-498a-9f8b-e72a3778bc7d" (UID: "2c913834-e613-498a-9f8b-e72a3778bc7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:05:36.460688 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.460673 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk" pod "2c913834-e613-498a-9f8b-e72a3778bc7d" (UID: "2c913834-e613-498a-9f8b-e72a3778bc7d"). InnerVolumeSpecName "kube-api-access-77xxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:05:36.559197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.559173 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77xxk\" (UniqueName: \"kubernetes.io/projected/2c913834-e613-498a-9f8b-e72a3778bc7d-kube-api-access-77xxk\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:05:36.559197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:36.559195 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c913834-e613-498a-9f8b-e72a3778bc7d-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:05:37.113235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.113204 4854 generic.go:356] "Generic (PLEG): container finished" podID="2c913834-e613-498a-9f8b-e72a3778bc7d" containerID="f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67" exitCode=0 Oct 31 03:05:37.113235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.113261 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" Oct 31 03:05:37.113702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.113285 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" event={"ID":"2c913834-e613-498a-9f8b-e72a3778bc7d","Type":"ContainerDied","Data":"f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67"} Oct 31 03:05:37.113702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.113354 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7" event={"ID":"2c913834-e613-498a-9f8b-e72a3778bc7d","Type":"ContainerDied","Data":"b609824423d6a26f95f08700acf0b8f0629a6a9a2214690fac8fa432f246709c"} Oct 31 03:05:37.113702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.113380 4854 scope.go:122] "RemoveContainer" containerID="f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67" Oct 31 03:05:37.122322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.122308 4854 scope.go:122] "RemoveContainer" containerID="f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67" Oct 31 03:05:37.122568 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:05:37.122550 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67\": container with ID starting with f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67 not found: ID does not exist" containerID="f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67" Oct 31 03:05:37.122626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.122575 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67"} err="failed to get container status \"f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67\": rpc error: code = NotFound desc = could not find container \"f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67\": container with ID starting with f53fabd4cdb79f66ad36feff45aeeab4dc2968f5f2b1fe5ce0d1b7c5d4f81d67 not found: ID does not exist" Oct 31 03:05:37.141899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.141876 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7"] Oct 31 03:05:37.144272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:37.144236 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-z2zz7"] Oct 31 03:05:38.455540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:05:38.455505 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c913834-e613-498a-9f8b-e72a3778bc7d" path="/var/lib/kubelet/pods/2c913834-e613-498a-9f8b-e72a3778bc7d/volumes" Oct 31 03:05:39.251008 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:05:39.250971 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:06:09.249856 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:06:09.249812 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:06:12.525997 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.525964 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm"] Oct 31 03:06:12.526515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.526362 4854 cpu_manager.go:414] "containerMap: removing container" podUID="2c913834-e613-498a-9f8b-e72a3778bc7d" containerName="manager" Oct 31 03:06:12.526515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.526373 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c913834-e613-498a-9f8b-e72a3778bc7d" containerName="manager" Oct 31 03:06:12.526515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.526470 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c913834-e613-498a-9f8b-e72a3778bc7d" containerName="manager" Oct 31 03:06:12.529988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.529967 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.545318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.545299 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-56afbd-openshift-default-dockercfg-xms2n\"" Oct 31 03:06:12.552515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.552490 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm"] Oct 31 03:06:12.627605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.627570 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4"] Oct 31 03:06:12.631133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.631116 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.642281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.642242 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-924ee5-openshift-default-dockercfg-w4fmh\"" Oct 31 03:06:12.645122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.645099 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4"] Oct 31 03:06:12.661428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661394 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661440 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661468 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661526 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l27m\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661630 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661676 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661759 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661788 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.661853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.661829 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.762413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762376 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.762413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762413 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.762638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762438 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xmt\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.762638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762561 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.762638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762608 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.762812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762680 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.762812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762714 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.762812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762759 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.762966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762872 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.762966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762901 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.762966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.762947 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763001 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.763122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763043 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763089 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763135 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.763335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763193 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.763335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763222 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763486 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763340 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763486 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763388 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763486 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763415 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l27m\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763486 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763426 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.763949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.763924 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.764161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.764143 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.765373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.765350 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.765475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.765358 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.781398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.781347 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l27m\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.783219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.783203 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.820194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.820157 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z"] Oct 31 03:06:12.825122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.825102 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.838231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.838205 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z"] Oct 31 03:06:12.839164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.839140 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-85cb59-openshift-default-dockercfg-fr8qt\"" Oct 31 03:06:12.844223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.844198 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:12.864638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.864607 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.864732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.864641 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.864732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.864686 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.864732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.864711 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.864741 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.864808 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.865016 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xmt\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.865117 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.865166 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.865681 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.865743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.865734 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.866594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.866562 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.866904 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.866861 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.867216 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.867192 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.869711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.869689 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.871361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.871338 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.886384 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.886356 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.889458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.888423 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xmt\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.947937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.947907 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:12.966768 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.966739 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.966862 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.966795 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.966908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.966876 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.966947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.966924 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.966994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.966972 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.967141 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.967116 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mpx\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.967187 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.967174 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.967268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.967217 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.967422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.967396 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:12.989488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.989461 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr"] Oct 31 03:06:12.994015 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:12.993973 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.011672 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.011647 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr"] Oct 31 03:06:13.016669 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.016643 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7d8daf-openshift-default-dockercfg-lm6g8\"" Oct 31 03:06:13.068943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.068874 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.068943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.068916 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mpx\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.068943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.068936 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.068985 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.069059 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.069090 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.069121 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.069483 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.069549 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.069739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.069582 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.070086 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.070064 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.070199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.070152 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.070324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.070224 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.071135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.071111 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.071729 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.071708 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.071833 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.071818 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.080503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.080483 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.080605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.080568 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mpx\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.141344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.141312 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:13.146268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.146153 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm"] Oct 31 03:06:13.148634 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:13.148608 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc78da3f_c4c7_4ff0_b398_388d0a07a04f.slice/crio-c4556cee2e8a3ea049e6bdac34def918b5a3aa6e4e00673977e0c0e7eeab0340 WatchSource:0}: Error finding container c4556cee2e8a3ea049e6bdac34def918b5a3aa6e4e00673977e0c0e7eeab0340: Status 404 returned error can't find the container with id c4556cee2e8a3ea049e6bdac34def918b5a3aa6e4e00673977e0c0e7eeab0340 Oct 31 03:06:13.159526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.158222 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.159526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.158341 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.159526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.158397 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.173700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.173662 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.174014 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.173993 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.174366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.174346 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.174557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.174538 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.174860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.174842 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.175151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.175132 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5hd\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.175405 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.175378 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.175863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.175834 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.175988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.175895 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.259519 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.258998 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4"] Oct 31 03:06:13.268573 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:13.268545 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba893f3_b888_41af_a479_93b4965a7813.slice/crio-8d91042f1fea54ef9d7a6edde54120d01d4935c6069d4f879f03051b7d7b7cb1 WatchSource:0}: Error finding container 8d91042f1fea54ef9d7a6edde54120d01d4935c6069d4f879f03051b7d7b7cb1: Status 404 returned error can't find the container with id 8d91042f1fea54ef9d7a6edde54120d01d4935c6069d4f879f03051b7d7b7cb1 Oct 31 03:06:13.271399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.271369 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.271506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.271433 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.271506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.271481 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.276863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.276833 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.276970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.276876 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.276970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.276916 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.276970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.276940 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5hd\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.277135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.276978 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.277135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.277009 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.277135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.277048 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.277135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.277089 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.277135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.277117 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.278193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.278169 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.279064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.279036 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.279714 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.279687 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.279943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.279923 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.280618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.280595 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.280721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.280685 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.281919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.281882 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.290633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.290562 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5hd\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.290879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.290859 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.296414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.296198 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" event={"ID":"fba893f3-b888-41af-a479-93b4965a7813","Type":"ContainerStarted","Data":"8d91042f1fea54ef9d7a6edde54120d01d4935c6069d4f879f03051b7d7b7cb1"} Oct 31 03:06:13.298473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.297743 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" event={"ID":"fc78da3f-c4c7-4ff0-b398-388d0a07a04f","Type":"ContainerStarted","Data":"ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af"} Oct 31 03:06:13.298473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.297784 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" event={"ID":"fc78da3f-c4c7-4ff0-b398-388d0a07a04f","Type":"ContainerStarted","Data":"c4556cee2e8a3ea049e6bdac34def918b5a3aa6e4e00673977e0c0e7eeab0340"} Oct 31 03:06:13.313477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.313224 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:13.486090 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.485606 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podStartSLOduration=1.485587658 podStartE2EDuration="1.485587658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:13.321702333 +0000 UTC m=+657.018869639" watchObservedRunningTime="2026-10-31 03:06:13.485587658 +0000 UTC m=+657.182754963" Oct 31 03:06:13.486090 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.485800 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z"] Oct 31 03:06:13.492113 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.492080 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.492220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.492143 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.492220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.492169 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.613157 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.613123 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr"] Oct 31 03:06:13.625268 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:13.625227 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dcb70a_fe35_4891_97c4_86b113224244.slice/crio-60e20a0465c00637193e4eb405eeebd4ce08ec1d19268b294e19b3065b1419f0 WatchSource:0}: Error finding container 60e20a0465c00637193e4eb405eeebd4ce08ec1d19268b294e19b3065b1419f0: Status 404 returned error can't find the container with id 60e20a0465c00637193e4eb405eeebd4ce08ec1d19268b294e19b3065b1419f0 Oct 31 03:06:13.629152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.629119 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.629218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.629204 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.629304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.629275 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:13.844884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:13.844805 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:14.304310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.304277 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" event={"ID":"fba893f3-b888-41af-a479-93b4965a7813","Type":"ContainerStarted","Data":"7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a"} Oct 31 03:06:14.306001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.305973 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" event={"ID":"9f386509-0bc2-4565-9b54-f4efa32fad2c","Type":"ContainerStarted","Data":"4f30fa83e164edb8bdc65afcf80c1065e2fec2edfc22e9f66e85f625f0da2f8f"} Oct 31 03:06:14.306143 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.306020 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" event={"ID":"9f386509-0bc2-4565-9b54-f4efa32fad2c","Type":"ContainerStarted","Data":"d03679ff86fa2d10f0b5f417473592d8c870781d1d93ba0c4bfefccbca66d079"} Oct 31 03:06:14.307667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.307642 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" event={"ID":"82dcb70a-fe35-4891-97c4-86b113224244","Type":"ContainerStarted","Data":"3acc7079f5e2ea72e968b98320f0b70893ba0d7ce7b6809b791ab29253a751fd"} Oct 31 03:06:14.307775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.307691 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" event={"ID":"82dcb70a-fe35-4891-97c4-86b113224244","Type":"ContainerStarted","Data":"60e20a0465c00637193e4eb405eeebd4ce08ec1d19268b294e19b3065b1419f0"} Oct 31 03:06:14.314132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.314116 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:14.319701 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.319680 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:14.329488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.329450 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podStartSLOduration=2.329437786 podStartE2EDuration="2.329437786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:14.327482192 +0000 UTC m=+658.024649495" watchObservedRunningTime="2026-10-31 03:06:14.329437786 +0000 UTC m=+658.026605090" Oct 31 03:06:14.351515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.351474 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podStartSLOduration=2.35146256 podStartE2EDuration="2.35146256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:14.349405656 +0000 UTC m=+658.046572961" watchObservedRunningTime="2026-10-31 03:06:14.35146256 +0000 UTC m=+658.048629869" Oct 31 03:06:14.371330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.371284 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podStartSLOduration=2.371273779 podStartE2EDuration="2.371273779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:14.369775961 +0000 UTC m=+658.066943262" watchObservedRunningTime="2026-10-31 03:06:14.371273779 +0000 UTC m=+658.068441080" Oct 31 03:06:14.850675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.850643 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:14.948090 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.948060 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:14.953333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:14.953305 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:15.142437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.142358 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:15.148002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.147978 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:15.312781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.312743 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:15.313182 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313074 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:15.313182 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313123 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:15.313182 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313150 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:15.313692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313671 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:15.313793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313736 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:15.313793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313782 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:15.313878 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:15.313835 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:17.709035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.709004 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r"] Oct 31 03:06:17.713936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.713918 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:17.731856 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.731833 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-56afbd\"" Oct 31 03:06:17.735305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.735284 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r"] Oct 31 03:06:17.735668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.735648 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-56afbd-dockercfg-x4qvx\"" Oct 31 03:06:17.741344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.741314 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:17.741456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.741369 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25v7w\" (UniqueName: \"kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w\") pod \"payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:17.842646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.842600 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:17.842646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.842653 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25v7w\" (UniqueName: \"kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w\") pod \"payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:17.843319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.843298 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:17.857715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:17.857683 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25v7w\" (UniqueName: \"kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w\") pod \"payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:18.031479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:18.031432 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:18.309095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:18.308949 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r"] Oct 31 03:06:18.314035 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:18.314005 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805261a2_7bf7_4b7e_8d55_28c65c030d58.slice/crio-ec5c8645c96c4bbe6ae5aa7dd22c4c85ebcb6944c6e98b12b96c376fb041e4e3 WatchSource:0}: Error finding container ec5c8645c96c4bbe6ae5aa7dd22c4c85ebcb6944c6e98b12b96c376fb041e4e3: Status 404 returned error can't find the container with id ec5c8645c96c4bbe6ae5aa7dd22c4c85ebcb6944c6e98b12b96c376fb041e4e3 Oct 31 03:06:18.332471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:18.332443 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" event={"ID":"805261a2-7bf7-4b7e-8d55-28c65c030d58","Type":"ContainerStarted","Data":"ec5c8645c96c4bbe6ae5aa7dd22c4c85ebcb6944c6e98b12b96c376fb041e4e3"} Oct 31 03:06:19.338800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.338759 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" event={"ID":"805261a2-7bf7-4b7e-8d55-28c65c030d58","Type":"ContainerStarted","Data":"9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726"} Oct 31 03:06:19.339242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.338926 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:19.358277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.358210 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" podStartSLOduration=2.358192472 podStartE2EDuration="2.358192472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:19.356127907 +0000 UTC m=+663.053295225" watchObservedRunningTime="2026-10-31 03:06:19.358192472 +0000 UTC m=+663.055359777" Oct 31 03:06:19.610866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.610761 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt"] Oct 31 03:06:19.614734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.614714 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.621702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.621680 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9"] Oct 31 03:06:19.622151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.622133 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-edb9e6\"" Oct 31 03:06:19.625368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.625348 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.627004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.626982 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt"] Oct 31 03:06:19.632963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.632942 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-edb9e6-dockercfg-9q2td\"" Oct 31 03:06:19.635584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.635561 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9"] Oct 31 03:06:19.667401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.667350 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wgs\" (UniqueName: \"kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs\") pod \"payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.667573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.667422 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.667573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.667558 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.667701 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.667600 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6z6q\" (UniqueName: \"kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q\") pod \"payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.779503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.779467 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.779844 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.779823 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.779959 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.779947 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z6q\" (UniqueName: \"kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q\") pod \"payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.780072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.780060 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wgs\" (UniqueName: \"kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs\") pod \"payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.780549 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.780519 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.781296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.781272 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.804343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.800903 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wgs\" (UniqueName: \"kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs\") pod \"payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:19.804343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.801376 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6z6q\" (UniqueName: \"kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q\") pod \"payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.928061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.927968 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:19.940290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:19.940262 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:20.218835 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:20.218789 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f93acb_8038_4ca5_a28e_a6b275ce8760.slice/crio-9c4c750bcd17e6709a756c5b61bf90d4a6a3637694b7dcef212ce0b26f3f2d37 WatchSource:0}: Error finding container 9c4c750bcd17e6709a756c5b61bf90d4a6a3637694b7dcef212ce0b26f3f2d37: Status 404 returned error can't find the container with id 9c4c750bcd17e6709a756c5b61bf90d4a6a3637694b7dcef212ce0b26f3f2d37 Oct 31 03:06:20.218835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.218799 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt"] Oct 31 03:06:20.236193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.236171 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9"] Oct 31 03:06:20.238283 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:20.238232 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76f6aae_6b69_4890_8556_20a3701f5f49.slice/crio-3993c3ecacd6a9dfdf60accb28de9b34189afd8dcd712e1c869ec5fe880cc664 WatchSource:0}: Error finding container 3993c3ecacd6a9dfdf60accb28de9b34189afd8dcd712e1c869ec5fe880cc664: Status 404 returned error can't find the container with id 3993c3ecacd6a9dfdf60accb28de9b34189afd8dcd712e1c869ec5fe880cc664 Oct 31 03:06:20.346907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.346875 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" event={"ID":"a76f6aae-6b69-4890-8556-20a3701f5f49","Type":"ContainerStarted","Data":"e729d68c2cc60bdd87b1c5d3b6e0f4658803a8757c091ee3aad1972bfc11bf7d"} Oct 31 03:06:20.347380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.346943 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" event={"ID":"a76f6aae-6b69-4890-8556-20a3701f5f49","Type":"ContainerStarted","Data":"3993c3ecacd6a9dfdf60accb28de9b34189afd8dcd712e1c869ec5fe880cc664"} Oct 31 03:06:20.347380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.347107 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:20.348610 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.348588 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" event={"ID":"81f93acb-8038-4ca5-a28e-a6b275ce8760","Type":"ContainerStarted","Data":"aad765d2c46678ea3f3b42ad4da24246d0d66383d2e0669de43fbac2f591adcc"} Oct 31 03:06:20.348785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.348769 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:20.348853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.348804 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" event={"ID":"81f93acb-8038-4ca5-a28e-a6b275ce8760","Type":"ContainerStarted","Data":"9c4c750bcd17e6709a756c5b61bf90d4a6a3637694b7dcef212ce0b26f3f2d37"} Oct 31 03:06:20.364263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.364213 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" podStartSLOduration=1.3641976470000001 podStartE2EDuration="1.364197647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.362942001 +0000 UTC m=+664.060109305" watchObservedRunningTime="2026-10-31 03:06:20.364197647 +0000 UTC m=+664.061364952" Oct 31 03:06:20.379117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.379075 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" podStartSLOduration=1.379063454 podStartE2EDuration="1.379063454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.378115404 +0000 UTC m=+664.075282708" watchObservedRunningTime="2026-10-31 03:06:20.379063454 +0000 UTC m=+664.076230758" Oct 31 03:06:20.887794 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.887761 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf"] Oct 31 03:06:20.891588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.891568 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:20.900630 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.900607 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7d8daf-dockercfg-xnz5c\"" Oct 31 03:06:20.902841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.902814 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf"] Oct 31 03:06:20.993974 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.993951 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7d8daf\"" Oct 31 03:06:20.998372 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.998351 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27dn\" (UniqueName: \"kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn\") pod \"payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:20.998471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:20.998400 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:21.099611 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.099572 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27dn\" (UniqueName: \"kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn\") pod \"payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:21.099785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.099646 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:21.100600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.100576 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:21.138994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.138918 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27dn\" (UniqueName: \"kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn\") pod \"payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:21.208897 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.208865 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:21.488966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.488938 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf"] Oct 31 03:06:21.491264 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:21.491218 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53edaf58_a577_4597_8c91_104af425b533.slice/crio-09829da88c70e0956f34a13faba5f29f6d3fe23b70f867d196a5fe8ca43f8aa3 WatchSource:0}: Error finding container 09829da88c70e0956f34a13faba5f29f6d3fe23b70f867d196a5fe8ca43f8aa3: Status 404 returned error can't find the container with id 09829da88c70e0956f34a13faba5f29f6d3fe23b70f867d196a5fe8ca43f8aa3 Oct 31 03:06:21.593657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.593629 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g"] Oct 31 03:06:21.597240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.597225 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.601326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.601304 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw"] Oct 31 03:06:21.604747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.604729 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.606835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.606823 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-85cb59\"" Oct 31 03:06:21.609422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.608964 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-85cb59-dockercfg-cnbvp\"" Oct 31 03:06:21.611509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.611489 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g"] Oct 31 03:06:21.619899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.619875 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw"] Oct 31 03:06:21.705170 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.705133 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.705368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.705241 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.705368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.705358 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rjn\" (UniqueName: \"kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn\") pod \"payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.705569 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.705544 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jl4\" (UniqueName: \"kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4\") pod \"payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.806587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.806511 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jl4\" (UniqueName: \"kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4\") pod \"payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.806587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.806566 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.806793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.806642 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.806793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.806685 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rjn\" (UniqueName: \"kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn\") pod \"payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.807565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.807541 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.807695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.807596 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.818113 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.818089 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rjn\" (UniqueName: \"kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn\") pod \"payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.818305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.818289 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jl4\" (UniqueName: \"kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4\") pod \"payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:21.910140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.910117 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:21.925919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:21.925901 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:22.144178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.143572 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2"] Oct 31 03:06:22.160180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.160144 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.163107 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.163064 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-bebea4-serving-cert\"" Oct 31 03:06:22.163240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.163229 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2"] Oct 31 03:06:22.195765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.195728 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g"] Oct 31 03:06:22.211484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.211463 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.211607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.211588 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.211675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.211662 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87f6\" (UniqueName: \"kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.211720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.211691 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.215575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.215555 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw"] Oct 31 03:06:22.217340 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:22.217308 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743857fb_f1c4_4cc7_a6c3_ba01a0e070cf.slice/crio-83ae60f57cebd6dd2eeeb4e506c6a65c43f2765b9594cbb5755a44144fd54cd6 WatchSource:0}: Error finding container 83ae60f57cebd6dd2eeeb4e506c6a65c43f2765b9594cbb5755a44144fd54cd6: Status 404 returned error can't find the container with id 83ae60f57cebd6dd2eeeb4e506c6a65c43f2765b9594cbb5755a44144fd54cd6 Oct 31 03:06:22.313416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.313383 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.313522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.313472 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87f6\" (UniqueName: \"kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.313597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.313519 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.313597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.313581 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.314215 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.314183 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.315995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.315971 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.316083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.315971 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.324383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.324361 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87f6\" (UniqueName: \"kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6\") pod \"maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.362616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.362588 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" event={"ID":"a847ac29-5362-45ac-9715-77a1c1a90e2b","Type":"ContainerStarted","Data":"32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f"} Oct 31 03:06:22.362736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.362648 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" event={"ID":"a847ac29-5362-45ac-9715-77a1c1a90e2b","Type":"ContainerStarted","Data":"3c38db24ad58913bfd1088381cbc94bc688d19321e2810fc3cf0406f8be5ac27"} Oct 31 03:06:22.362736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.362680 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:22.364428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.364403 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" event={"ID":"53edaf58-a577-4597-8c91-104af425b533","Type":"ContainerStarted","Data":"720efe4ea433af077d2bfdcb86e208009fe5e12a9dbaf5128bc32bc75664c237"} Oct 31 03:06:22.364537 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.364455 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:22.364537 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.364481 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" event={"ID":"53edaf58-a577-4597-8c91-104af425b533","Type":"ContainerStarted","Data":"09829da88c70e0956f34a13faba5f29f6d3fe23b70f867d196a5fe8ca43f8aa3"} Oct 31 03:06:22.366065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.366045 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" event={"ID":"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf","Type":"ContainerStarted","Data":"7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf"} Oct 31 03:06:22.366166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.366079 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" event={"ID":"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf","Type":"ContainerStarted","Data":"83ae60f57cebd6dd2eeeb4e506c6a65c43f2765b9594cbb5755a44144fd54cd6"} Oct 31 03:06:22.366166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.366112 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:22.379687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.379639 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" podStartSLOduration=1.379625913 podStartE2EDuration="1.379625913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:22.378098403 +0000 UTC m=+666.075265708" watchObservedRunningTime="2026-10-31 03:06:22.379625913 +0000 UTC m=+666.076793218" Oct 31 03:06:22.396200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.396126 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" podStartSLOduration=2.396113503 podStartE2EDuration="2.396113503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:22.393594502 +0000 UTC m=+666.090761808" watchObservedRunningTime="2026-10-31 03:06:22.396113503 +0000 UTC m=+666.093280810" Oct 31 03:06:22.409902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.409868 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" podStartSLOduration=1.409858203 podStartE2EDuration="1.409858203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:22.409750579 +0000 UTC m=+666.106917883" watchObservedRunningTime="2026-10-31 03:06:22.409858203 +0000 UTC m=+666.107025507" Oct 31 03:06:22.481969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.481931 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:22.779572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.779546 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2"] Oct 31 03:06:22.781442 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:22.781417 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63bffd9_d5ca_4f28_8b98_42379e1df918.slice/crio-7ec439cc7d7440da8dc8a2d689044af252cd02b79c343bc5caf5ae044127023e WatchSource:0}: Error finding container 7ec439cc7d7440da8dc8a2d689044af252cd02b79c343bc5caf5ae044127023e: Status 404 returned error can't find the container with id 7ec439cc7d7440da8dc8a2d689044af252cd02b79c343bc5caf5ae044127023e Oct 31 03:06:22.782913 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:22.782897 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:06:23.372633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:23.372601 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" event={"ID":"f63bffd9-d5ca-4f28-8b98-42379e1df918","Type":"ContainerStarted","Data":"7ec439cc7d7440da8dc8a2d689044af252cd02b79c343bc5caf5ae044127023e"} Oct 31 03:06:24.380339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.380305 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" event={"ID":"f63bffd9-d5ca-4f28-8b98-42379e1df918","Type":"ContainerStarted","Data":"e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8"} Oct 31 03:06:24.380984 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.380967 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:24.406956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.406891 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" podStartSLOduration=1.925782406 podStartE2EDuration="2.406873354s" totalImagesPullingTime="481.090948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:24.404625209 +0000 UTC m=+668.101792555" watchObservedRunningTime="2026-10-31 03:06:24.406873354 +0000 UTC m=+668.104040659" Oct 31 03:06:24.845921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.845886 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd"] Oct 31 03:06:24.851454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.851422 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd"] Oct 31 03:06:24.851650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.851435 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:24.854318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.854286 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-924ee5-serving-cert\"" Oct 31 03:06:24.983707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.983673 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr298\" (UniqueName: \"kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:24.983865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.983827 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:24.983941 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.983883 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:24.983996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:24.983961 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.085022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.084981 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.085201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.085033 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.085201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.085139 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr298\" (UniqueName: \"kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.085297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.085227 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.085785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.085757 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.087681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.087660 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.088000 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.087981 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.111305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.102875 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr298\" (UniqueName: \"kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298\") pod \"maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.170092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.170053 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:25.531275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:25.531046 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd"] Oct 31 03:06:25.567091 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:25.567057 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf4c889_9afb_4325_b343_ec7fb2251357.slice/crio-bc9915c2c2cd672f2084802798abfa3a0fada385c3f992abb418d71d6218c24c WatchSource:0}: Error finding container bc9915c2c2cd672f2084802798abfa3a0fada385c3f992abb418d71d6218c24c: Status 404 returned error can't find the container with id bc9915c2c2cd672f2084802798abfa3a0fada385c3f992abb418d71d6218c24c Oct 31 03:06:26.412562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:26.412308 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" event={"ID":"9bf4c889-9afb-4325-b343-ec7fb2251357","Type":"ContainerStarted","Data":"4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50"} Oct 31 03:06:26.412562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:26.412381 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" event={"ID":"9bf4c889-9afb-4325-b343-ec7fb2251357","Type":"ContainerStarted","Data":"bc9915c2c2cd672f2084802798abfa3a0fada385c3f992abb418d71d6218c24c"} Oct 31 03:06:26.412562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:26.412430 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:26.458063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:26.458013 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" podStartSLOduration=2.003746231 podStartE2EDuration="2.457999846s" totalImagesPullingTime="454.253615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:26.456369765 +0000 UTC m=+670.153537071" watchObservedRunningTime="2026-10-31 03:06:26.457999846 +0000 UTC m=+670.155167151" Oct 31 03:06:30.360273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:30.356285 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:06:31.358824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.358791 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:06:31.358986 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.358886 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:06:31.359411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.359389 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:06:31.359524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.359438 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:06:31.359524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.359390 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:06:31.359631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.359552 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:06:31.435033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:31.434318 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:06:32.456475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:32.456444 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:06:33.377292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.376379 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:06:33.378282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.377776 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:06:33.378718 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.378700 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:06:33.485683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.485650 4854 generic.go:356] "Generic (PLEG): container finished" podID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerID="4f30fa83e164edb8bdc65afcf80c1065e2fec2edfc22e9f66e85f625f0da2f8f" exitCode=0 Oct 31 03:06:33.486155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.485873 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" event={"ID":"9f386509-0bc2-4565-9b54-f4efa32fad2c","Type":"ContainerDied","Data":"4f30fa83e164edb8bdc65afcf80c1065e2fec2edfc22e9f66e85f625f0da2f8f"} Oct 31 03:06:33.486313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.486292 4854 scope.go:122] "RemoveContainer" containerID="4f30fa83e164edb8bdc65afcf80c1065e2fec2edfc22e9f66e85f625f0da2f8f" Oct 31 03:06:33.487590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.487565 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:33.487794 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.487773 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:33.487892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.487815 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:33.488231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.488161 4854 generic.go:356] "Generic (PLEG): container finished" podID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerID="ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af" exitCode=0 Oct 31 03:06:33.488231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.488203 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" event={"ID":"fc78da3f-c4c7-4ff0-b398-388d0a07a04f","Type":"ContainerDied","Data":"ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af"} Oct 31 03:06:33.488573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.488556 4854 scope.go:122] "RemoveContainer" containerID="ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af" Oct 31 03:06:33.525273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.489800 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:33.537882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.525564 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:33.546441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:33.546398 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:34.546631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:34.546593 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" event={"ID":"fc78da3f-c4c7-4ff0-b398-388d0a07a04f","Type":"ContainerStarted","Data":"1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345"} Oct 31 03:06:34.565404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:34.565370 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" event={"ID":"9f386509-0bc2-4565-9b54-f4efa32fad2c","Type":"ContainerStarted","Data":"99c981e9504b87c3b84ca973013e0e0675a7801d0bd1c5cfa066247bb611578d"} Oct 31 03:06:34.850333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:34.849350 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:34.854272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:34.853411 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:06:34.854272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:34.853475 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:06:35.150325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:35.142119 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:35.150325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:35.142736 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:06:35.150325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:35.142814 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 03:06:35.848106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:35.847657 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:06:35.848106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:35.847719 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:06:36.146335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:36.145806 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:06:36.146335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:36.145895 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 03:06:36.852275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:36.851803 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:06:36.852275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:36.851875 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:06:37.149351 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:37.145634 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:06:37.149351 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:37.145695 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 03:06:37.849719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:37.849619 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:06:37.849719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:37.849670 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:06:38.151732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:38.150301 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:06:38.151732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:38.150363 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 03:06:38.844743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:38.844692 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:06:38.844949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:38.844758 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:06:39.142707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:39.142599 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:06:39.142707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:39.142669 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 03:06:39.282494 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:06:39.282431 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:06:39.845097 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:39.845058 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 03:06:39.845325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:39.845111 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 03:06:40.142879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:40.142776 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:06:40.142879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:40.142854 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 03:06:41.146783 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:41.146743 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:41.147306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:41.147066 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:41.147820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:41.147797 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:06:41.178124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:41.178089 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:41.179766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:41.179747 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:41.180783 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:41.180762 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:06:42.701115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:42.701018 4854 generic.go:356] "Generic (PLEG): container finished" podID="82dcb70a-fe35-4891-97c4-86b113224244" containerID="3acc7079f5e2ea72e968b98320f0b70893ba0d7ce7b6809b791ab29253a751fd" exitCode=0 Oct 31 03:06:42.704148 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:42.701612 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" event={"ID":"82dcb70a-fe35-4891-97c4-86b113224244","Type":"ContainerDied","Data":"3acc7079f5e2ea72e968b98320f0b70893ba0d7ce7b6809b791ab29253a751fd"} Oct 31 03:06:42.704148 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:42.702731 4854 scope.go:122] "RemoveContainer" containerID="3acc7079f5e2ea72e968b98320f0b70893ba0d7ce7b6809b791ab29253a751fd" Oct 31 03:06:42.704683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:42.704531 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:42.704683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:42.704621 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:42.704683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:42.704667 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:43.717937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.717899 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" event={"ID":"82dcb70a-fe35-4891-97c4-86b113224244","Type":"ContainerStarted","Data":"fb259279580f2f9e0a24a084755b728bd857d9ae149bc2780d7c877684ec3792"} Oct 31 03:06:43.719574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.719550 4854 generic.go:356] "Generic (PLEG): container finished" podID="fba893f3-b888-41af-a479-93b4965a7813" containerID="7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a" exitCode=0 Oct 31 03:06:43.719693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.719602 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" event={"ID":"fba893f3-b888-41af-a479-93b4965a7813","Type":"ContainerDied","Data":"7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a"} Oct 31 03:06:43.719905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.719892 4854 scope.go:122] "RemoveContainer" containerID="7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a" Oct 31 03:06:43.721237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.721202 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:43.721361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.721290 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:43.721361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:43.721347 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:06:44.314419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.314382 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:44.315910 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.315880 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:06:44.316044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.315943 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:06:44.744295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.744232 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" event={"ID":"fba893f3-b888-41af-a479-93b4965a7813","Type":"ContainerStarted","Data":"d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544"} Oct 31 03:06:44.949377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.949347 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:44.950380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.950350 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 03:06:44.950614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:44.950584 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 03:06:45.314374 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:45.314341 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:45.315026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:45.315005 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:45.315305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:45.315207 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:06:45.315466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:45.315444 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:06:45.951516 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:45.951476 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 03:06:45.951979 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:45.951539 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 03:06:46.329166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:46.329130 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:06:46.329496 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:46.329466 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:06:46.953279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:46.952449 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 03:06:46.953279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:46.952523 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 03:06:47.318274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:47.317694 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:06:47.318274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:47.317764 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:06:47.960279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:47.951030 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 03:06:47.960279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:47.951097 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 03:06:48.317275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:48.316708 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 03:06:48.317275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:48.316777 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 03:06:48.949199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:48.949162 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 03:06:48.949431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:48.949226 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 03:06:49.324950 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:49.324920 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:49.335580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:49.335553 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:06:49.949145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:49.949112 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 03:06:49.949348 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:49.949160 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 03:06:50.952942 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:50.952914 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:50.953877 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:50.953747 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:06:56.660567 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.660537 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn"] Oct 31 03:06:56.672260 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.672223 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.678098 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.678056 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"default-dockercfg-76j64\"" Oct 31 03:06:56.679129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.679104 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn"] Oct 31 03:06:56.684142 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.684122 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"kube-root-ca.crt\"" Oct 31 03:06:56.684327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.684147 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"openshift-service-ca.crt\"" Oct 31 03:06:56.697939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.697920 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"simulator-w0-edb9e6-kserve-self-signed-certs\"" Oct 31 03:06:56.809583 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.809558 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.809713 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.809594 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.809713 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.809619 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5t28\" (UniqueName: \"kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.809801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.809753 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.809801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.809772 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.809897 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.809864 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911215 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911338 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911376 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911399 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5t28\" (UniqueName: \"kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911467 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911492 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911789 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.911882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.911866 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.913301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.913275 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.914056 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.914027 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.914165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.914143 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.926446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.926418 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5t28\" (UniqueName: \"kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28\") pod \"simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:56.985684 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:56.985663 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:06:57.277714 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:57.277691 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn"] Oct 31 03:06:57.280309 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:06:57.280282 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc8fd16b_51b4_47eb_ad4f_7226961f85f8.slice/crio-e55640668fcc2dfc429b770a9365cb545d4c3759dca026a544e39dd93c2a668b WatchSource:0}: Error finding container e55640668fcc2dfc429b770a9365cb545d4c3759dca026a544e39dd93c2a668b: Status 404 returned error can't find the container with id e55640668fcc2dfc429b770a9365cb545d4c3759dca026a544e39dd93c2a668b Oct 31 03:06:57.823970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:57.823935 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" event={"ID":"bc8fd16b-51b4-47eb-ad4f-7226961f85f8","Type":"ContainerStarted","Data":"a8079303242ca717008cc7a02c2fa23211a28027ea77f6dd36eaae08bd31f474"} Oct 31 03:06:57.823970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:57.823973 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" event={"ID":"bc8fd16b-51b4-47eb-ad4f-7226961f85f8","Type":"ContainerStarted","Data":"e55640668fcc2dfc429b770a9365cb545d4c3759dca026a544e39dd93c2a668b"} Oct 31 03:06:57.855810 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:57.855382 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podStartSLOduration=1.85535977 podStartE2EDuration="1.85535977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:57.854671002 +0000 UTC m=+701.551838306" watchObservedRunningTime="2026-10-31 03:06:57.85535977 +0000 UTC m=+701.552527074" Oct 31 03:06:57.986501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:06:57.986399 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:07:06.986731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:06.986695 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:07:06.999027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:06.999003 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:07:07.898329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:07.898295 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:07:09.249750 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:07:09.249715 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:07:09.431369 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.431340 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m"] Oct 31 03:07:09.436867 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.436836 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.450744 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.450721 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"openshift-service-ca.crt\"" Oct 31 03:07:09.451108 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.451081 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"kube-root-ca.crt\"" Oct 31 03:07:09.452996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.452975 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"default-dockercfg-gdkj4\"" Oct 31 03:07:09.452996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.452979 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"premium-w5-56afbd-kserve-self-signed-certs\"" Oct 31 03:07:09.453947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.453927 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m"] Oct 31 03:07:09.558371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.558289 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.558658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.558630 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.558931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.558895 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvn7l\" (UniqueName: \"kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.559118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.559102 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.559203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.559183 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.559464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.559440 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.660600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.660566 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.660775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.660618 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.660775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.660658 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.660775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.660684 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.660775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.660710 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.661002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.660775 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvn7l\" (UniqueName: \"kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.661102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.661083 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.661165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.661112 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.661165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.661131 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.663572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.663547 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.663805 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.663789 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.673787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.673762 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvn7l\" (UniqueName: \"kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l\") pod \"premium-w5-56afbd-kserve-5468b7768c-s7j4m\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:09.752556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:09.752530 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:10.035046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:10.035019 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m"] Oct 31 03:07:10.036778 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:07:10.036750 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d6bd592_ed4c_4a14_9455_fbdd18419ad8.slice/crio-12ef4df9f1e47c912e883e1380b7f8108143fba87296584d04c3da423d04f2a5 WatchSource:0}: Error finding container 12ef4df9f1e47c912e883e1380b7f8108143fba87296584d04c3da423d04f2a5: Status 404 returned error can't find the container with id 12ef4df9f1e47c912e883e1380b7f8108143fba87296584d04c3da423d04f2a5 Oct 31 03:07:10.902838 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:10.902800 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" event={"ID":"9d6bd592-ed4c-4a14-9455-fbdd18419ad8","Type":"ContainerStarted","Data":"a3beddf22dfee12a7dacafff7bbaf19b3165b4b35acb7612f7c072a0e4422aab"} Oct 31 03:07:10.903312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:10.902853 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" event={"ID":"9d6bd592-ed4c-4a14-9455-fbdd18419ad8","Type":"ContainerStarted","Data":"12ef4df9f1e47c912e883e1380b7f8108143fba87296584d04c3da423d04f2a5"} Oct 31 03:07:10.925063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:10.925019 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podStartSLOduration=1.925007436 podStartE2EDuration="1.925007436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:10.922834694 +0000 UTC m=+714.620002020" watchObservedRunningTime="2026-10-31 03:07:10.925007436 +0000 UTC m=+714.622174740" Oct 31 03:07:11.753200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:11.753168 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:19.753271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.753226 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:19.757028 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.757002 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh"] Oct 31 03:07:19.762436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.762413 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.766367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.766344 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"kube-root-ca.crt\"" Oct 31 03:07:19.769238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.769216 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"openshift-service-ca.crt\"" Oct 31 03:07:19.769969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.769601 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"default-dockercfg-2rmv8\"" Oct 31 03:07:19.770061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.769744 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"premium-w3-99bc97-kserve-self-signed-certs\"" Oct 31 03:07:19.770061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.769867 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:19.774959 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.774940 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh"] Oct 31 03:07:19.899471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.899435 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.899811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.899791 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.901224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.900044 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.901455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.901435 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.901594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.901576 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhc2\" (UniqueName: \"kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.901846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.901819 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:19.982048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:19.982021 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:07:20.002845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.002653 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003073 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003152 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhc2\" (UniqueName: \"kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003227 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003307 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003352 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003492 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003686 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.003823 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.003692 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.005709 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.005686 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.006145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.006120 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.015259 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.015218 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhc2\" (UniqueName: \"kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2\") pod \"premium-w3-99bc97-kserve-6974d5db44-kpvdh\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.076001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.075976 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:20.373571 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.373547 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh"] Oct 31 03:07:20.375448 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:07:20.375420 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49c944be_03b8_4015_9126_1eac4954734f.slice/crio-7bd66895ad076a0b91fdfd1879d7c5515135165a586c8aaec239c7808e5d76fa WatchSource:0}: Error finding container 7bd66895ad076a0b91fdfd1879d7c5515135165a586c8aaec239c7808e5d76fa: Status 404 returned error can't find the container with id 7bd66895ad076a0b91fdfd1879d7c5515135165a586c8aaec239c7808e5d76fa Oct 31 03:07:20.978769 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.978735 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" event={"ID":"49c944be-03b8-4015-9126-1eac4954734f","Type":"ContainerStarted","Data":"2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b"} Oct 31 03:07:20.979209 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.978780 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" event={"ID":"49c944be-03b8-4015-9126-1eac4954734f","Type":"ContainerStarted","Data":"7bd66895ad076a0b91fdfd1879d7c5515135165a586c8aaec239c7808e5d76fa"} Oct 31 03:07:20.998349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:20.998305 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podStartSLOduration=1.998292047 podStartE2EDuration="1.998292047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:20.996570676 +0000 UTC m=+724.693737980" watchObservedRunningTime="2026-10-31 03:07:20.998292047 +0000 UTC m=+724.695459348" Oct 31 03:07:21.076650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:21.076612 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:28.327360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.327326 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94"] Oct 31 03:07:28.332358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.332334 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.338944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.338917 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"openshift-service-ca.crt\"" Oct 31 03:07:28.339188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.339169 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"kube-root-ca.crt\"" Oct 31 03:07:28.339308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.339237 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"premium-w6-7d8daf-kserve-self-signed-certs\"" Oct 31 03:07:28.339308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.339291 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"default-dockercfg-zjnj2\"" Oct 31 03:07:28.346290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.346265 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94"] Oct 31 03:07:28.406505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.406470 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.406666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.406616 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.406666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.406634 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.406743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.406703 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zgfb\" (UniqueName: \"kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.406743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.406737 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.406805 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.406774 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513037 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zgfb\" (UniqueName: \"kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513093 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513154 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513243 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513431 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513459 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.513693 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.514675 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.515511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.515464 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.518700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.518674 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.521167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.521147 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.526857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.526836 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zgfb\" (UniqueName: \"kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb\") pod \"premium-w6-7d8daf-kserve-66d8bc4788-k7g94\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.648346 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.648241 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:28.927535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:28.927510 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94"] Oct 31 03:07:28.928840 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:07:28.928814 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42347c06_7df7_4f5c_acd3_9127f8f42302.slice/crio-97825ec1a092366c200174a0a2d7fb9f5ae2d5f9900225277c21d1060b32e1e0 WatchSource:0}: Error finding container 97825ec1a092366c200174a0a2d7fb9f5ae2d5f9900225277c21d1060b32e1e0: Status 404 returned error can't find the container with id 97825ec1a092366c200174a0a2d7fb9f5ae2d5f9900225277c21d1060b32e1e0 Oct 31 03:07:29.033480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:29.033439 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" event={"ID":"42347c06-7df7-4f5c-acd3-9127f8f42302","Type":"ContainerStarted","Data":"c70c5dca47babe5f916b0fb1510450cca7ac055be845466a4be5fdefc7724f29"} Oct 31 03:07:29.033657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:29.033496 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" event={"ID":"42347c06-7df7-4f5c-acd3-9127f8f42302","Type":"ContainerStarted","Data":"97825ec1a092366c200174a0a2d7fb9f5ae2d5f9900225277c21d1060b32e1e0"} Oct 31 03:07:29.056001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:29.055951 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podStartSLOduration=1.055936569 podStartE2EDuration="1.055936569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:29.052897485 +0000 UTC m=+732.750064811" watchObservedRunningTime="2026-10-31 03:07:29.055936569 +0000 UTC m=+732.753103873" Oct 31 03:07:29.648510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:29.648483 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:30.077178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:30.077147 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:30.089434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:30.089411 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:31.054963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:31.054929 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:07:38.649210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:38.649173 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:38.662037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:38.662016 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:39.108051 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:39.108014 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:07:39.288676 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:07:39.288636 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:07:42.574857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.574825 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm"] Oct 31 03:07:42.578842 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.578827 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.583882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.583845 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bebea4-openshift-default-dockercfg-wg8mv\"" Oct 31 03:07:42.592385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.592363 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm"] Oct 31 03:07:42.703087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703058 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rk4r\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703089 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703115 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703177 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703345 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703369 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703386 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703545 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.703628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.703603 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.804704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804675 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.804855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804724 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.804855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804755 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rk4r\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.804855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804780 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.804855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804820 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.804855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804846 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804940 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804968 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.804994 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.805414 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805797 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.805642 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.805640 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.805667 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.805895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.805829 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.807212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.807192 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.807422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.807404 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.816078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.816055 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.816206 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.816190 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rk4r\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:42.894139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:42.894085 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:43.113057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.113025 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr"] Oct 31 03:07:43.117219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.117199 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.120091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.120065 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-edb9e6-openshift-default-dockercfg-n4zc9\"" Oct 31 03:07:43.130597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.130575 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr"] Oct 31 03:07:43.177894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.177875 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm"] Oct 31 03:07:43.180349 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:07:43.180324 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cac45d_e413_4766_84e7_50b4ec39edde.slice/crio-a991ef504f7a9322b72411f68d2bc3a17c5c3be15f66a2e65d15d87450921733 WatchSource:0}: Error finding container a991ef504f7a9322b72411f68d2bc3a17c5c3be15f66a2e65d15d87450921733: Status 404 returned error can't find the container with id a991ef504f7a9322b72411f68d2bc3a17c5c3be15f66a2e65d15d87450921733 Oct 31 03:07:43.182901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.182867 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:43.182965 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.182948 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:43.183009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.182991 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:43.309580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309544 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.309757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309615 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.309818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309751 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.309818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309795 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.309931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309873 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4bm\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.309931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309915 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.310034 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.309994 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.310091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.310076 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.310155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.310132 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411239 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411304 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411417 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4bm\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411459 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411501 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411557 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411614 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411656 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411723 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.411920 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411901 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.412001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.411978 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.412171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.412140 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.412337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.412195 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.412397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.412337 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.413989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.413971 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.414398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.414383 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.423281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.423219 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.423654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.423633 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4bm\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.431294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.431227 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:43.731948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.731922 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr"] Oct 31 03:07:43.733889 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:07:43.733861 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390e34b_2306_40bf_b38e_54da40de46b8.slice/crio-e1114d1ca829f6943fbc44c3152c55cf0d066d8ecb12cb83fe32548ce630d915 WatchSource:0}: Error finding container e1114d1ca829f6943fbc44c3152c55cf0d066d8ecb12cb83fe32548ce630d915: Status 404 returned error can't find the container with id e1114d1ca829f6943fbc44c3152c55cf0d066d8ecb12cb83fe32548ce630d915 Oct 31 03:07:43.736380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.736352 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:43.736455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.736420 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:43.736455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:43.736449 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:44.127467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.127433 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" event={"ID":"e5cac45d-e413-4766-84e7-50b4ec39edde","Type":"ContainerStarted","Data":"1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1"} Oct 31 03:07:44.127670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.127493 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" event={"ID":"e5cac45d-e413-4766-84e7-50b4ec39edde","Type":"ContainerStarted","Data":"a991ef504f7a9322b72411f68d2bc3a17c5c3be15f66a2e65d15d87450921733"} Oct 31 03:07:44.129463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.129431 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" event={"ID":"8390e34b-2306-40bf-b38e-54da40de46b8","Type":"ContainerStarted","Data":"9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987"} Oct 31 03:07:44.129564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.129484 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" event={"ID":"8390e34b-2306-40bf-b38e-54da40de46b8","Type":"ContainerStarted","Data":"e1114d1ca829f6943fbc44c3152c55cf0d066d8ecb12cb83fe32548ce630d915"} Oct 31 03:07:44.149578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.149514 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podStartSLOduration=2.14949834 podStartE2EDuration="2.14949834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:44.148095343 +0000 UTC m=+747.845262649" watchObservedRunningTime="2026-10-31 03:07:44.14949834 +0000 UTC m=+747.846665650" Oct 31 03:07:44.169586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.169531 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podStartSLOduration=1.169516484 podStartE2EDuration="1.169516484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:44.168773826 +0000 UTC m=+747.865941127" watchObservedRunningTime="2026-10-31 03:07:44.169516484 +0000 UTC m=+747.866683788" Oct 31 03:07:44.432333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.431671 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:44.439276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.438572 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:44.439276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.438687 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:44.896040 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.895341 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:44.916275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.909381 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:07:44.916275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:44.909448 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:07:45.437291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:45.436653 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:45.437291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:45.436713 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:45.905541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:45.905494 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:07:45.906571 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:45.906538 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:07:46.436274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:46.435568 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:46.436274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:46.435625 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:46.908273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:46.898277 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:07:46.908273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:46.898333 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:07:47.435275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:47.431611 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:47.435275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:47.431671 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:47.899287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:47.897547 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:07:47.899287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:47.897607 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:07:48.432281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:48.432214 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:48.433444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:48.433411 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:48.905805 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:48.905770 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:07:48.906101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:48.906072 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:07:49.432665 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:49.432629 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:49.433346 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:49.433314 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:49.897233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:49.897190 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:07:49.897494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:49.897314 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 03:07:50.431697 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:50.431662 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:07:50.431876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:50.431723 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 03:07:50.899284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:50.899235 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:50.899752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:50.899673 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:50.900523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:50.900502 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:07:51.455590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:51.455556 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:51.456067 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:51.456047 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:51.457071 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:51.457032 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:07:57.895540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:57.895460 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs"] Oct 31 03:07:57.900072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:57.900051 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:57.907673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:57.907649 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-99bc97-openshift-default-dockercfg-pzvq9\"" Oct 31 03:07:57.912007 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:57.911987 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs"] Oct 31 03:07:58.009658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009631 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.009793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009664 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.009793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009750 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.009793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009788 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.009908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009812 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446fj\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.009943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009923 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.009978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.009964 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.010022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.010010 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.010130 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.010110 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.110876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.110850 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.110880 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.110913 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.110936 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.110959 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446fj\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.110994 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111020 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111082 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111148 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111292 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111334 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111492 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111445 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111605 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.111939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.111918 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.118228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.113739 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.118228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.114033 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.146844 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.146773 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.148658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.146865 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446fj\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.217167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.217135 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:07:58.504522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.504497 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs"] Oct 31 03:07:58.506284 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:07:58.506261 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb86aec6_406a_44f5_b7a6_01876224e495.slice/crio-db346bbbec07dbd17797fb8659aad032568105935755a559e3ee6a4428db5d66 WatchSource:0}: Error finding container db346bbbec07dbd17797fb8659aad032568105935755a559e3ee6a4428db5d66: Status 404 returned error can't find the container with id db346bbbec07dbd17797fb8659aad032568105935755a559e3ee6a4428db5d66 Oct 31 03:07:58.508493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.508467 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:58.508561 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.508525 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:58.508561 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:58.508553 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:07:59.259401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:59.259363 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" event={"ID":"db86aec6-406a-44f5-b7a6-01876224e495","Type":"ContainerStarted","Data":"4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c"} Oct 31 03:07:59.259835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:59.259441 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" event={"ID":"db86aec6-406a-44f5-b7a6-01876224e495","Type":"ContainerStarted","Data":"db346bbbec07dbd17797fb8659aad032568105935755a559e3ee6a4428db5d66"} Oct 31 03:07:59.282208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:07:59.282162 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podStartSLOduration=2.282149517 podStartE2EDuration="2.282149517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:59.280003936 +0000 UTC m=+762.977171240" watchObservedRunningTime="2026-10-31 03:07:59.282149517 +0000 UTC m=+762.979316855" Oct 31 03:08:00.218192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:00.218156 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:08:00.219955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:00.219929 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:08:00.220078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:00.219984 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:08:01.218051 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:01.218014 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:08:01.218508 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:01.218075 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:08:02.217567 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:02.217523 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:08:02.217744 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:02.217615 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:08:03.218339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:03.218304 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:08:03.218738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:03.218366 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:08:04.218145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:04.218103 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:08:04.218409 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:04.218181 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 03:08:05.232667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:05.228566 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:08:05.232667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:05.229947 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:08:05.233205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:05.233010 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:08:09.251060 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:08:09.251020 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:08:21.256397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.256368 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn"] Oct 31 03:08:21.261775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.261754 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.272013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.271990 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:21.272142 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.272084 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:21.274920 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.274899 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn"] Oct 31 03:08:21.303358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.303326 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.303477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.303412 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4nj\" (UniqueName: \"kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.303565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.303540 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.404640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.404609 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.404781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.404707 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.404781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.404763 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4nj\" (UniqueName: \"kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.405268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.405231 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.407538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.407516 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.416585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.416562 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4nj\" (UniqueName: \"kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj\") pod \"maas-api-revoke-sub-aab747d9-lcdbn\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.582787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.582419 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:21.887575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:21.887550 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn"] Oct 31 03:08:21.890074 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:08:21.890021 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c3cc8d_0353_4d49_9afc_e2e30bdc54bf.slice/crio-285d1528439fd5ed48a0e4c1248e7c4c60e8e0ab69906c6158c0258db2561e65 WatchSource:0}: Error finding container 285d1528439fd5ed48a0e4c1248e7c4c60e8e0ab69906c6158c0258db2561e65: Status 404 returned error can't find the container with id 285d1528439fd5ed48a0e4c1248e7c4c60e8e0ab69906c6158c0258db2561e65 Oct 31 03:08:22.424673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:22.424646 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" event={"ID":"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf","Type":"ContainerStarted","Data":"285d1528439fd5ed48a0e4c1248e7c4c60e8e0ab69906c6158c0258db2561e65"} Oct 31 03:08:25.443023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:25.442993 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" event={"ID":"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf","Type":"ContainerStarted","Data":"25fb39d46701e582934e88ddede8a0c28574dbbca56172de865a06353c05f517"} Oct 31 03:08:25.461222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:25.460631 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" podStartSLOduration=1.213929161 podStartE2EDuration="4.460614635s" totalImagesPullingTime="3.246685474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:25.458865219 +0000 UTC m=+789.156032522" watchObservedRunningTime="2026-10-31 03:08:25.460614635 +0000 UTC m=+789.157781939" Oct 31 03:08:26.515276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:26.514637 4854 generic.go:356] "Generic (PLEG): container finished" podID="c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" containerID="25fb39d46701e582934e88ddede8a0c28574dbbca56172de865a06353c05f517" exitCode=0 Oct 31 03:08:26.515276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:26.514752 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" event={"ID":"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf","Type":"ContainerDied","Data":"25fb39d46701e582934e88ddede8a0c28574dbbca56172de865a06353c05f517"} Oct 31 03:08:27.683067 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.683043 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:27.845159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.845076 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj\" (UniqueName: \"kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj\") pod \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " Oct 31 03:08:27.845408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.845160 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca\") pod \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " Oct 31 03:08:27.845408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.845241 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca\") pod \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\" (UID: \"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf\") " Oct 31 03:08:27.846381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.846351 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca" pod "c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" (UID: "c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:27.848494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.848463 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca" pod "c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" (UID: "c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:27.848606 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.848580 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj" pod "c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" (UID: "c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf"). InnerVolumeSpecName "kube-api-access-xl4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:27.947323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.947292 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4nj\" (UniqueName: \"kubernetes.io/projected/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-kube-api-access-xl4nj\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:27.947323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.947324 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-xks-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:27.947495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:27.947340 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf-maas-api-openshift-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:28.527842 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:28.527813 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" Oct 31 03:08:28.527842 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:28.527831 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn" event={"ID":"c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf","Type":"ContainerDied","Data":"285d1528439fd5ed48a0e4c1248e7c4c60e8e0ab69906c6158c0258db2561e65"} Oct 31 03:08:28.528024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:28.527878 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285d1528439fd5ed48a0e4c1248e7c4c60e8e0ab69906c6158c0258db2561e65" Oct 31 03:08:31.588289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.588238 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt"] Oct 31 03:08:31.589044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.589028 4854 cpu_manager.go:414] "containerMap: removing container" podUID="c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" containerName="revoke-keys" Oct 31 03:08:31.589089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.589048 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" containerName="revoke-keys" Oct 31 03:08:31.589212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.589201 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" containerName="revoke-keys" Oct 31 03:08:31.593225 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.593205 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.597828 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.597807 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:31.597948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.597922 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:31.601574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.601555 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt"] Oct 31 03:08:31.688772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.688745 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.688908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.688776 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkjv\" (UniqueName: \"kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.688949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.688904 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.790414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.790383 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.790557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.790466 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.790557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.790484 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkjv\" (UniqueName: \"kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.791303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.791280 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.816629 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.816596 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.819617 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.819586 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkjv\" (UniqueName: \"kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv\") pod \"maas-api-revoke-sub-a327e88e-g66bt\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:31.911489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:31.911425 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:32.188626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:32.188601 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt"] Oct 31 03:08:32.190841 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:08:32.190814 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa71a56_ee52_4860_b3d2_aadd712aa40b.slice/crio-e072a5d620ea2d6ab0f4a059041f65cacdf3c070fceb68a4bb49cdd61ffbbed7 WatchSource:0}: Error finding container e072a5d620ea2d6ab0f4a059041f65cacdf3c070fceb68a4bb49cdd61ffbbed7: Status 404 returned error can't find the container with id e072a5d620ea2d6ab0f4a059041f65cacdf3c070fceb68a4bb49cdd61ffbbed7 Oct 31 03:08:32.551133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:32.551097 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" event={"ID":"3fa71a56-ee52-4860-b3d2-aadd712aa40b","Type":"ContainerStarted","Data":"6bede7a02733466d6fba4b702a3751fd99234bf8cbcc64c976979606de156bcb"} Oct 31 03:08:32.551280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:32.551158 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" event={"ID":"3fa71a56-ee52-4860-b3d2-aadd712aa40b","Type":"ContainerStarted","Data":"e072a5d620ea2d6ab0f4a059041f65cacdf3c070fceb68a4bb49cdd61ffbbed7"} Oct 31 03:08:32.569732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:32.569688 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" podStartSLOduration=1.569674129 podStartE2EDuration="1.569674129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:32.566764609 +0000 UTC m=+796.263931911" watchObservedRunningTime="2026-10-31 03:08:32.569674129 +0000 UTC m=+796.266841443" Oct 31 03:08:33.557490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:33.557457 4854 generic.go:356] "Generic (PLEG): container finished" podID="3fa71a56-ee52-4860-b3d2-aadd712aa40b" containerID="6bede7a02733466d6fba4b702a3751fd99234bf8cbcc64c976979606de156bcb" exitCode=0 Oct 31 03:08:33.557905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:33.557538 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" event={"ID":"3fa71a56-ee52-4860-b3d2-aadd712aa40b","Type":"ContainerDied","Data":"6bede7a02733466d6fba4b702a3751fd99234bf8cbcc64c976979606de156bcb"} Oct 31 03:08:35.183485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.183420 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:35.268654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.268622 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca\") pod \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " Oct 31 03:08:35.268815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.268767 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca\") pod \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " Oct 31 03:08:35.268872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.268822 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv\" (UniqueName: \"kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv\") pod \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\" (UID: \"3fa71a56-ee52-4860-b3d2-aadd712aa40b\") " Oct 31 03:08:35.269041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.269020 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca" pod "3fa71a56-ee52-4860-b3d2-aadd712aa40b" (UID: "3fa71a56-ee52-4860-b3d2-aadd712aa40b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:35.269279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.269224 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-openshift-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:35.270952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.270928 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv" pod "3fa71a56-ee52-4860-b3d2-aadd712aa40b" (UID: "3fa71a56-ee52-4860-b3d2-aadd712aa40b"). InnerVolumeSpecName "kube-api-access-8nkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:35.271059 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.271022 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca" pod "3fa71a56-ee52-4860-b3d2-aadd712aa40b" (UID: "3fa71a56-ee52-4860-b3d2-aadd712aa40b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:35.379280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.377765 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa71a56-ee52-4860-b3d2-aadd712aa40b-maas-api-xks-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:35.379280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.378096 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nkjv\" (UniqueName: \"kubernetes.io/projected/3fa71a56-ee52-4860-b3d2-aadd712aa40b-kube-api-access-8nkjv\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:35.596960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.596926 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" Oct 31 03:08:35.597134 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.596970 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt" event={"ID":"3fa71a56-ee52-4860-b3d2-aadd712aa40b","Type":"ContainerDied","Data":"e072a5d620ea2d6ab0f4a059041f65cacdf3c070fceb68a4bb49cdd61ffbbed7"} Oct 31 03:08:35.597134 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:35.597016 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e072a5d620ea2d6ab0f4a059041f65cacdf3c070fceb68a4bb49cdd61ffbbed7" Oct 31 03:08:39.264189 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:08:39.264148 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:08:42.683482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.683437 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98"] Oct 31 03:08:42.684243 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.684218 4854 cpu_manager.go:414] "containerMap: removing container" podUID="3fa71a56-ee52-4860-b3d2-aadd712aa40b" containerName="revoke-keys" Oct 31 03:08:42.684458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.684432 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa71a56-ee52-4860-b3d2-aadd712aa40b" containerName="revoke-keys" Oct 31 03:08:42.684641 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.684627 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa71a56-ee52-4860-b3d2-aadd712aa40b" containerName="revoke-keys" Oct 31 03:08:42.688753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.688728 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.702364 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.702328 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98"] Oct 31 03:08:42.786070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786040 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786209 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786084 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786201 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786233 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786348 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786311 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5lr\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786405 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786391 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786422 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786458 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.786525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.786502 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887470 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5lr\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887665 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887512 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887690 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887738 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887829 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887817 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887890 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.887973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.887953 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888056 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888060 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888128 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888303 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888315 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888521 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.888623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.888512 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.890404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.890379 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.890516 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.890498 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.919266 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.919231 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5lr\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:42.920008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:42.919985 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:43.005506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.005477 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:43.301464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.301434 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98"] Oct 31 03:08:43.303100 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:08:43.303068 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2d1aac_33ad_47e9_a872_6b33e04b386e.slice/crio-3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da WatchSource:0}: Error finding container 3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da: Status 404 returned error can't find the container with id 3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da Oct 31 03:08:43.306082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.306045 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:08:43.306167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.306132 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:08:43.306208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.306172 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:08:43.679472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.679397 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" event={"ID":"9b2d1aac-33ad-47e9-a872-6b33e04b386e","Type":"ContainerStarted","Data":"5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255"} Oct 31 03:08:43.679472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.679450 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" event={"ID":"9b2d1aac-33ad-47e9-a872-6b33e04b386e","Type":"ContainerStarted","Data":"3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da"} Oct 31 03:08:43.706019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:43.705968 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podStartSLOduration=1.705953863 podStartE2EDuration="1.705953863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:43.701426402 +0000 UTC m=+807.398593707" watchObservedRunningTime="2026-10-31 03:08:43.705953863 +0000 UTC m=+807.403121166" Oct 31 03:08:44.009933 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:44.009893 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:44.012371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:44.012334 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:44.012932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:44.012620 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:45.014276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:45.013624 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:45.014276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:45.013688 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:46.010274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:46.009570 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:46.010274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:46.009625 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:47.016707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:47.016665 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.017217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:47.016726 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:48.044274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:48.043374 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:48.044274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:48.043431 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:48.984363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:48.984317 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:49.012240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:49.012204 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.012588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:49.012559 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:50.008295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:50.008239 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:50.008885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:50.008856 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 03:08:51.737347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:51.737317 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:51.737826 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:51.737594 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:08:51.738721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:08:51.738689 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:09:02.112568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.112534 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf"] Oct 31 03:09:02.116529 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.116510 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.138310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.138287 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf"] Oct 31 03:09:02.140646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.140621 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj"] Oct 31 03:09:02.148494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.148470 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.168430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.168403 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj"] Oct 31 03:09:02.221682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221655 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.221813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221687 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.221813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221720 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.221813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221792 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmwt\" (UniqueName: \"kubernetes.io/projected/5facc9ef-64c0-46b2-8d88-75e73722a799-kube-api-access-tsmwt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.221996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221841 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.221996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221921 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.221996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.221987 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.222106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222017 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.222106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222065 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rt9\" (UniqueName: \"kubernetes.io/projected/28f78da7-dfdb-43d8-8062-86bb0e8daba1-kube-api-access-k4rt9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.222173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222112 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.222173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222155 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5facc9ef-64c0-46b2-8d88-75e73722a799-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.222263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222174 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.222263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222222 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.222379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222274 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.222379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222334 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.222379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222363 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.222509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222452 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.222509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.222473 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323090 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323051 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323090 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323087 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323112 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323129 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323168 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323190 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmwt\" (UniqueName: \"kubernetes.io/projected/5facc9ef-64c0-46b2-8d88-75e73722a799-kube-api-access-tsmwt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323221 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323280 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323334 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323359 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323384 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rt9\" (UniqueName: \"kubernetes.io/projected/28f78da7-dfdb-43d8-8062-86bb0e8daba1-kube-api-access-k4rt9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323534 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.323699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323658 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323665 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.323699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323700 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323756 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5facc9ef-64c0-46b2-8d88-75e73722a799-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323785 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323815 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323844 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323868 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323885 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323898 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.323998 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.324424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.324211 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.324424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.324270 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.324296 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.324877 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.324853 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5facc9ef-64c0-46b2-8d88-75e73722a799-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.325027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.324924 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.326647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.326620 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.326754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.326652 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.326754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.326677 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.327069 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.327053 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.336160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.336097 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f78da7-dfdb-43d8-8062-86bb0e8daba1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.338005 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.337951 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rt9\" (UniqueName: \"kubernetes.io/projected/28f78da7-dfdb-43d8-8062-86bb0e8daba1-kube-api-access-k4rt9\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7v4sj\" (UID: \"28f78da7-dfdb-43d8-8062-86bb0e8daba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.338455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.338430 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5facc9ef-64c0-46b2-8d88-75e73722a799-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.338584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.338505 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmwt\" (UniqueName: \"kubernetes.io/projected/5facc9ef-64c0-46b2-8d88-75e73722a799-kube-api-access-tsmwt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lp4mf\" (UID: \"5facc9ef-64c0-46b2-8d88-75e73722a799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.436263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.433551 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:02.471001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.470975 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:02.729064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.729040 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf"] Oct 31 03:09:02.731048 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:02.731014 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5facc9ef_64c0_46b2_8d88_75e73722a799.slice/crio-964538a79d32aa9c6904f2548d8178ea1eed7b9979813869107cd69b8e10e287 WatchSource:0}: Error finding container 964538a79d32aa9c6904f2548d8178ea1eed7b9979813869107cd69b8e10e287: Status 404 returned error can't find the container with id 964538a79d32aa9c6904f2548d8178ea1eed7b9979813869107cd69b8e10e287 Oct 31 03:09:02.733542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.733513 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:02.733621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.733575 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:02.733621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.733605 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:02.781532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.781505 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj"] Oct 31 03:09:02.784913 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:02.784878 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f78da7_dfdb_43d8_8062_86bb0e8daba1.slice/crio-70b6e1624fe995e3363147b06459e39c19eb7e1e8fd567234720f9c2e414a684 WatchSource:0}: Error finding container 70b6e1624fe995e3363147b06459e39c19eb7e1e8fd567234720f9c2e414a684: Status 404 returned error can't find the container with id 70b6e1624fe995e3363147b06459e39c19eb7e1e8fd567234720f9c2e414a684 Oct 31 03:09:02.787799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.787757 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:02.787882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.787844 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:02.787922 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.787892 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:02.924267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.923716 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" event={"ID":"5facc9ef-64c0-46b2-8d88-75e73722a799","Type":"ContainerStarted","Data":"af2bf61da1219b3dd3cdbb3416e9d8d7bfbaf91f131165d67af0ab0d027b84f7"} Oct 31 03:09:02.924267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.923774 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" event={"ID":"5facc9ef-64c0-46b2-8d88-75e73722a799","Type":"ContainerStarted","Data":"964538a79d32aa9c6904f2548d8178ea1eed7b9979813869107cd69b8e10e287"} Oct 31 03:09:02.930422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.930392 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" event={"ID":"28f78da7-dfdb-43d8-8062-86bb0e8daba1","Type":"ContainerStarted","Data":"388a000be441a92f9ad86e744d6f4a3dec9dbfff506b2a521ae8a01ed521eaaf"} Oct 31 03:09:02.930533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.930448 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" event={"ID":"28f78da7-dfdb-43d8-8062-86bb0e8daba1","Type":"ContainerStarted","Data":"70b6e1624fe995e3363147b06459e39c19eb7e1e8fd567234720f9c2e414a684"} Oct 31 03:09:02.981003 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.980896 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podStartSLOduration=0.980871541 podStartE2EDuration="980.871541ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:02.959088657 +0000 UTC m=+826.656256009" watchObservedRunningTime="2026-10-31 03:09:02.980871541 +0000 UTC m=+826.678038845" Oct 31 03:09:02.984488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:02.984433 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podStartSLOduration=0.984416132 podStartE2EDuration="984.416132ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:02.979665448 +0000 UTC m=+826.676832751" watchObservedRunningTime="2026-10-31 03:09:02.984416132 +0000 UTC m=+826.681583443" Oct 31 03:09:03.439456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:03.439363 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:03.439947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:03.439538 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:03.439947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:03.439598 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:03.472187 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:03.472156 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:03.473605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:03.473584 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:03.473656 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:03.473628 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:04.436547 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:04.436511 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:04.494313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:04.493882 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:04.494960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:04.494935 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:04.495151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:04.495128 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:05.439291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:05.438639 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:05.439291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:05.438698 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:05.473276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:05.472685 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:05.473276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:05.472757 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:06.439275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:06.438383 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:06.439275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:06.438449 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:06.487651 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:06.487057 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:06.487651 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:06.487117 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:07.446274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:07.437613 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:07.446274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:07.437675 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:07.474621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:07.474573 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:07.474804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:07.474645 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:08.438116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.438078 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:08.438478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.438435 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:08.472040 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.472005 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:08.472438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.472066 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:08.663472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.663436 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww"] Oct 31 03:09:08.682151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.682109 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww"] Oct 31 03:09:08.682342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.682223 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:08.688642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.688576 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"default-dockercfg-6w94r\"" Oct 31 03:09:08.689052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.688820 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"distinct-w1-924ee5-kserve-self-signed-certs\"" Oct 31 03:09:08.694505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.694485 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"openshift-service-ca.crt\"" Oct 31 03:09:08.721273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.695325 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"kube-root-ca.crt\"" Oct 31 03:09:08.869477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.869436 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:08.896375 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.896334 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6mm\" (UniqueName: \"kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:08.896620 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.896604 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:08.896721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.896709 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:08.896902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.896888 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:08.897074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:08.897057 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.000983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.000941 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.001236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.001215 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6mm\" (UniqueName: \"kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.001403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.001388 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.001517 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.001504 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.001693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.001680 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.001789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.001777 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.007306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.006181 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.007306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.006476 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.012289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.011018 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.058129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.058096 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6mm\" (UniqueName: \"kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.071387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.059019 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.071816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.071795 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm\") pod \"distinct-w1-924ee5-kserve-84f9644db-kfhww\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.271622 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:09:09.271534 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:09:09.318740 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.318431 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:09.451022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.443663 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:09.451022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.443721 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:09.471992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.471957 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:09.472154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.472034 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:09.660075 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:09.660052 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww"] Oct 31 03:09:09.661873 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:09.661846 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c17918e_87b0_4d8c_88e5_d072a50b9f59.slice/crio-554d579a087953bf812c676649cb06d4032e11b44e9878baf30a64bc2c269504 WatchSource:0}: Error finding container 554d579a087953bf812c676649cb06d4032e11b44e9878baf30a64bc2c269504: Status 404 returned error can't find the container with id 554d579a087953bf812c676649cb06d4032e11b44e9878baf30a64bc2c269504 Oct 31 03:09:10.246779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.246735 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" event={"ID":"1c17918e-87b0-4d8c-88e5-d072a50b9f59","Type":"ContainerStarted","Data":"4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b"} Oct 31 03:09:10.246779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.246784 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" event={"ID":"1c17918e-87b0-4d8c-88e5-d072a50b9f59","Type":"ContainerStarted","Data":"554d579a087953bf812c676649cb06d4032e11b44e9878baf30a64bc2c269504"} Oct 31 03:09:10.265285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.265220 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podStartSLOduration=2.2652079990000002 podStartE2EDuration="2.265207999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:10.263589507 +0000 UTC m=+833.960756821" watchObservedRunningTime="2026-10-31 03:09:10.265207999 +0000 UTC m=+833.962375340" Oct 31 03:09:10.319138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.319114 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:10.436642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.436603 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:10.436806 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.436667 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:10.472350 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.472320 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:10.472879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:10.472373 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:11.436827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:11.436787 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 03:09:11.437009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:11.436851 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 03:09:11.471972 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:11.471944 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 03:09:11.472114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:11.471978 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 03:09:12.551686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:12.551638 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:12.553158 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:12.553128 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:12.553961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:12.553944 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" Oct 31 03:09:12.609921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:12.609885 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:12.610530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:12.610509 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:12.611698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:12.611664 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" Oct 31 03:09:18.285915 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.285877 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc"] Oct 31 03:09:18.291173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.291151 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.294289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.294228 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"default-dockercfg-cwqcr\"" Oct 31 03:09:18.294466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.294318 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"openshift-service-ca.crt\"" Oct 31 03:09:18.294466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.294335 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"kube-root-ca.crt\"" Oct 31 03:09:18.294801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.294779 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"distinct-w2-85cb59-kserve-self-signed-certs\"" Oct 31 03:09:18.301643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.301622 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc"] Oct 31 03:09:18.400642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.400612 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l2z8\" (UniqueName: \"kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.400802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.400711 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.400802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.400758 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.400802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.400801 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.400924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.400909 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.400991 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.400975 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502374 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502466 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l2z8\" (UniqueName: \"kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502525 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502575 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502633 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502680 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502838 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502813 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.502923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502901 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.503010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.502980 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.504981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.504955 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.505157 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.505139 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.513646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.513617 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l2z8\" (UniqueName: \"kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8\") pod \"distinct-w2-85cb59-kserve-66cccf77dc-vflkc\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.608578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.608492 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:18.914848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:18.914821 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc"] Oct 31 03:09:18.921808 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:18.921574 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5ecbffa_bea3_48b4_aa3d_5ab7e14603a0.slice/crio-1064ecca76861f3cebbdd280d558955234233d71586ae90f4428d35ff32030a0 WatchSource:0}: Error finding container 1064ecca76861f3cebbdd280d558955234233d71586ae90f4428d35ff32030a0: Status 404 returned error can't find the container with id 1064ecca76861f3cebbdd280d558955234233d71586ae90f4428d35ff32030a0 Oct 31 03:09:19.319037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:19.318997 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:19.321331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:19.321296 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" event={"ID":"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0","Type":"ContainerStarted","Data":"9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d"} Oct 31 03:09:19.321471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:19.321343 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" event={"ID":"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0","Type":"ContainerStarted","Data":"1064ecca76861f3cebbdd280d558955234233d71586ae90f4428d35ff32030a0"} Oct 31 03:09:19.332447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:19.332427 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:19.339292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:19.339202 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podStartSLOduration=1.3391889510000001 podStartE2EDuration="1.339188951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:19.338855722 +0000 UTC m=+843.036023026" watchObservedRunningTime="2026-10-31 03:09:19.339188951 +0000 UTC m=+843.036356249" Oct 31 03:09:19.609795 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:19.609708 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:20.360356 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:20.360319 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:09:27.945999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:27.945911 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq"] Oct 31 03:09:28.005624 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.005574 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq"] Oct 31 03:09:28.005796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.005632 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.093888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.093855 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094051 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.093933 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094051 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.093957 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bln\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.094118 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.094280 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.094305 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.094325 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.094414 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.094458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.094448 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.106726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.106699 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf"] Oct 31 03:09:28.110988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.110966 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.128484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.128456 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf"] Oct 31 03:09:28.195620 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195591 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.195750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195631 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.195750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195658 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.195750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195718 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.195887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195771 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.195887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195800 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.195887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195834 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.195887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195883 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195911 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bln\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195937 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.195970 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196029 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196059 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196094 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196220 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196291 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196334 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196334 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196407 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196412 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.196681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196515 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196548 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkvp\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.196818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.196795 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.198430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.198412 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.198991 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.198972 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.207868 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.207845 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.208095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.208077 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bln\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.297533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297501 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.297733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297647 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.297733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297682 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.297869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297736 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.297869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297771 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.297869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297857 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297889 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297937 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.297964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.298002 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkvp\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.298045 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.298194 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.298516 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.298631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.298612 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.301479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.301456 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.301723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.301703 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.320896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.320869 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:28.524358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.523713 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.537535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.537488 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkvp\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.548144 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.548118 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf"] Oct 31 03:09:28.553177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.553159 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf"] Oct 31 03:09:28.553292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.553263 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555340 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4f4\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555474 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555389 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555559 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555541 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555571 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555658 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555746 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555798 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555846 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.555932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.555917 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.608723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.608701 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq"] Oct 31 03:09:28.609658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.609625 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:28.610328 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:28.610304 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb898ea0f_f027_4065_bab3_cedbab0d2975.slice/crio-400629ca0e50982efb2a47869f28767cbfb5b4d06d399b625c31b9f6a330c6a2 WatchSource:0}: Error finding container 400629ca0e50982efb2a47869f28767cbfb5b4d06d399b625c31b9f6a330c6a2: Status 404 returned error can't find the container with id 400629ca0e50982efb2a47869f28767cbfb5b4d06d399b625c31b9f6a330c6a2 Oct 31 03:09:28.612841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.612814 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:28.612926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.612886 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:28.612926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.612914 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:28.622718 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.622697 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:28.656963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.656939 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.656988 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657012 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657044 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4f4\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657093 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657289 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657327 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657406 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657419 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657429 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657631 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657716 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657703 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.657968 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.657936 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.658420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.658399 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.660171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.660148 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.660348 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.660327 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.668306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.668285 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.668445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.668429 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4f4\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:28.732839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.732808 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:28.873333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:28.873304 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:29.021582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.020359 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf"] Oct 31 03:09:29.024931 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:29.024887 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8d3bba_a73b_4ac8_a0bc_4b56415346f2.slice/crio-6aae9c198194f5576ec71ebc2019fb365c0fa5423e9b1ae1bd6adbf99c4e2b09 WatchSource:0}: Error finding container 6aae9c198194f5576ec71ebc2019fb365c0fa5423e9b1ae1bd6adbf99c4e2b09: Status 404 returned error can't find the container with id 6aae9c198194f5576ec71ebc2019fb365c0fa5423e9b1ae1bd6adbf99c4e2b09 Oct 31 03:09:29.027402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.027367 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:29.027489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.027429 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:29.027489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.027460 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:29.222199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.222148 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf"] Oct 31 03:09:29.262662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.261189 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:29.262662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.261306 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:29.262662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.261354 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:29.569944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.569903 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" event={"ID":"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2","Type":"ContainerStarted","Data":"3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a"} Oct 31 03:09:29.570113 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.569994 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" event={"ID":"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2","Type":"ContainerStarted","Data":"6aae9c198194f5576ec71ebc2019fb365c0fa5423e9b1ae1bd6adbf99c4e2b09"} Oct 31 03:09:29.571823 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.571786 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" event={"ID":"a463dafe-3ebd-4c43-9376-934853ff4b36","Type":"ContainerStarted","Data":"6747f426c24665ff2025e9b26a90fd1769cef739167284d6c398fd954f403e0a"} Oct 31 03:09:29.572002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.571985 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" event={"ID":"a463dafe-3ebd-4c43-9376-934853ff4b36","Type":"ContainerStarted","Data":"636dfa5bcf5c508168e264b74cf4b4cf1e4e2b6eae33b9b9bc1e14fe3dea644b"} Oct 31 03:09:29.573763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.573745 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" event={"ID":"b898ea0f-f027-4065-bab3-cedbab0d2975","Type":"ContainerStarted","Data":"ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2"} Oct 31 03:09:29.573907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.573895 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" event={"ID":"b898ea0f-f027-4065-bab3-cedbab0d2975","Type":"ContainerStarted","Data":"400629ca0e50982efb2a47869f28767cbfb5b4d06d399b625c31b9f6a330c6a2"} Oct 31 03:09:29.586376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.586326 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:09:29.627437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.627376 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podStartSLOduration=1.627355184 podStartE2EDuration="1.627355184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.613780415 +0000 UTC m=+853.310947722" watchObservedRunningTime="2026-10-31 03:09:29.627355184 +0000 UTC m=+853.324522489" Oct 31 03:09:29.740317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.735378 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:29.740317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.740272 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podStartSLOduration=1.7402335 podStartE2EDuration="1.7402335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.715116923 +0000 UTC m=+853.412284228" watchObservedRunningTime="2026-10-31 03:09:29.7402335 +0000 UTC m=+853.437400805" Oct 31 03:09:29.748380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.746692 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podStartSLOduration=2.746677479 podStartE2EDuration="2.746677479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.743741814 +0000 UTC m=+853.440909118" watchObservedRunningTime="2026-10-31 03:09:29.746677479 +0000 UTC m=+853.443844783" Oct 31 03:09:29.873754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:29.873657 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:30.323505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.323471 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:30.340473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.340438 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.342566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.342529 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 03:09:30.736328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.736292 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.736607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.736577 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 03:09:30.875646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.875609 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.875846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:30.875660 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 03:09:31.324641 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:31.324602 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.325315 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:31.325212 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 03:09:31.736734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:31.736697 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.737971 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:31.737941 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 03:09:31.873744 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:31.873700 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.873925 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:31.873759 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 03:09:32.323274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:32.322678 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.323274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:32.322742 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 03:09:32.735780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:32.735746 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.736395 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:32.736348 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 03:09:32.873772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:32.873738 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.873958 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:32.873794 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 03:09:33.324274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:33.322428 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.324274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:33.322488 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 03:09:33.734328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:33.733669 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.734328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:33.733731 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 03:09:33.875273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:33.874625 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.875273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:33.874676 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 03:09:34.323126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:34.322675 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.323126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:34.322738 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 03:09:34.733731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:34.733637 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.734021 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:34.733992 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 03:09:34.876281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:34.875858 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.876281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:34.875920 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 03:09:35.321634 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:35.321594 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.321932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:35.321902 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 03:09:35.733423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:35.733326 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.733423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:35.733396 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 03:09:35.873791 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:35.873753 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.873971 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:35.873806 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 03:09:36.495261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:36.495211 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:36.495698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:36.495666 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:36.496517 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:36.496493 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:09:36.758682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:36.758605 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:36.762032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:36.760152 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:36.762032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:36.761956 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:09:37.081077 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:37.080994 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:37.081578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:37.081555 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:37.082703 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:37.082680 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:09:39.302898 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:09:39.302854 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:09:41.106888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.106859 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg"] Oct 31 03:09:41.112635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.112610 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.115903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.115877 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"distinct-2-w1-924ee5-kserve-self-signed-certs\"" Oct 31 03:09:41.126093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.126072 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg"] Oct 31 03:09:41.163650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.163614 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.163905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.163886 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgr7\" (UniqueName: \"kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.163993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.163979 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.164030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.164015 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.164066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.164037 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.164129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.164114 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.283964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgr7\" (UniqueName: \"kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.284070 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.284105 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.284143 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.284186 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.284274 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.286270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.284965 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.291516 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.289635 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.331323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.300632 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.331323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.306539 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.331323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.315074 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgr7\" (UniqueName: \"kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.336726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.336660 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs\") pod \"distinct-2-w1-924ee5-kserve-558c887945-4qgmg\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.429460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.429386 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:41.731092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.731043 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg"] Oct 31 03:09:41.791560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:41.790838 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" event={"ID":"327587c2-603d-4c5e-8c05-f06f49e634f9","Type":"ContainerStarted","Data":"14529a6c86d5ea283ab92601309d46d66c877288a332abe4fc2fdf78f88a5e20"} Oct 31 03:09:42.797489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:42.797456 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8"] Oct 31 03:09:42.802793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:42.802768 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:42.804490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:42.804457 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" event={"ID":"327587c2-603d-4c5e-8c05-f06f49e634f9","Type":"ContainerStarted","Data":"671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9"} Oct 31 03:09:42.832203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:42.832171 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8"] Oct 31 03:09:42.982280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:42.979373 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podStartSLOduration=1.979353216 podStartE2EDuration="1.979353216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:42.978661369 +0000 UTC m=+866.675828673" watchObservedRunningTime="2026-10-31 03:09:42.979353216 +0000 UTC m=+866.676520521" Oct 31 03:09:43.043578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.043544 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj"] Oct 31 03:09:43.074737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.074655 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.095355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.095311 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.095802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.095688 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.103297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.097712 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.104284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.104034 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj"] Oct 31 03:09:43.117874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.117839 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.167432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.167389 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttf4\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.168159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.168136 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.184331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.168644 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.184789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.184666 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.191658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.190365 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.204498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.204473 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m"] Oct 31 03:09:43.212141 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.212106 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m"] Oct 31 03:09:43.212316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.212240 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.291914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.291878 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.292089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.291931 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.292089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.291962 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.292089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292054 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.292191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292114 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttf4\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.292191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292148 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.292295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292194 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.292295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292225 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.292295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292273 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.292394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292299 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.292394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292381 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.292546 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292517 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.292615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292588 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.292705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292687 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxhc\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.292825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292749 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.292825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292797 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.292939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292866 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.292939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292913 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.293047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.292999 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293062 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.293100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293065 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293145 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293183 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55gm\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.293330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293209 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293239 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.293330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293309 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293339 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293355 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.293488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293372 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.293488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293305 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293678 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293525 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.293734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.293717 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.295886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.295860 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.295992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.295904 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.304184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.304157 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.304441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.304419 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttf4\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.395111 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395006 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395111 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395074 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395112 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxhc\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395143 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395190 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395274 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395299 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55gm\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395340 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395386 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395413 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395514 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395539 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395562 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395625 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395675 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395702 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395739 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.395792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.395763 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.397592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.396635 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.397592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.396919 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.397592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.397267 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.398707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.397615 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.398707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.398400 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.400353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.398241 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.400353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.399173 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.400353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.399443 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.400353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.399912 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.400742 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.400716 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.402509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.402484 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.402617 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.402594 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.403167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.403133 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.403368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.403349 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.408990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.408969 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxhc\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.412287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.412240 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.413536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.413512 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:43.414335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.414313 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55gm\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.433816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.429463 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:43.433816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.429753 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:43.512075 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.511769 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:43.539184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:43.538936 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:44.312029 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:44.298099 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m"] Oct 31 03:09:44.485458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:44.485417 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:44.490125 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:44.490090 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:44.507101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:44.507048 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:45.122350 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.122317 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8"] Oct 31 03:09:45.343107 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.318394 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" event={"ID":"17a8a06b-ea59-4eaa-b20f-99bde34963a0","Type":"ContainerStarted","Data":"a1d8de4df6b4fec1ee0987d550a0c046540a6f351605b128c73ba6e10eb10b61"} Oct 31 03:09:45.408301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.406525 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:45.408301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.406683 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:45.408301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.406773 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:45.660465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.660427 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:45.799394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.781517 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj"] Oct 31 03:09:45.854273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.854127 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:45.854273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.854226 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:45.854481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:45.854293 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:09:46.325638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.325590 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" event={"ID":"17a8a06b-ea59-4eaa-b20f-99bde34963a0","Type":"ContainerStarted","Data":"40e6cad9a2033b9dc41a884579d9affcf0f67f42c133913925fdda001952c8a2"} Oct 31 03:09:46.328573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.328539 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" event={"ID":"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8","Type":"ContainerStarted","Data":"03a16040f8fd1834935dbfc20cc0fcc31821c15a1ded986b82a675814f8505c8"} Oct 31 03:09:46.330116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.330059 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" event={"ID":"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8","Type":"ContainerStarted","Data":"850d22080068a6565a9fa323deb151ae9d4d5af63788c195b5c623dd0477544c"} Oct 31 03:09:46.332705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.332682 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" event={"ID":"a8122523-cf19-4951-8e8c-54dec0ee3aca","Type":"ContainerStarted","Data":"4a225f5387ca2836a8d973b4c828d127be6c7b3f0e227fdbcf2fa5493a8c95fc"} Oct 31 03:09:46.332869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.332852 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" event={"ID":"a8122523-cf19-4951-8e8c-54dec0ee3aca","Type":"ContainerStarted","Data":"59e1018217146a0bca55484d55a99eb9c1e09d52328d63b11ab3146cb5d5f47a"} Oct 31 03:09:46.348457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.348408 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podStartSLOduration=3.348391961 podStartE2EDuration="3.348391961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.346794862 +0000 UTC m=+870.043962168" watchObservedRunningTime="2026-10-31 03:09:46.348391961 +0000 UTC m=+870.045559266" Oct 31 03:09:46.376277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.376196 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podStartSLOduration=3.3761632710000002 podStartE2EDuration="3.376163271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.372985407 +0000 UTC m=+870.070152712" watchObservedRunningTime="2026-10-31 03:09:46.376163271 +0000 UTC m=+870.073330576" Oct 31 03:09:46.403467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.403412 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podStartSLOduration=4.403393051 podStartE2EDuration="4.403393051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.397583041 +0000 UTC m=+870.094750347" watchObservedRunningTime="2026-10-31 03:09:46.403393051 +0000 UTC m=+870.100560336" Oct 31 03:09:46.430402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.430358 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:46.513373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.513335 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:46.540434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:46.540405 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:47.434024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:47.433990 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.434648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:47.434616 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:47.518278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:47.517459 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.518278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:47.517522 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 03:09:47.547274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:47.546712 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.547274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:47.546770 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:09:48.439275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:48.438751 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.439275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:48.438819 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:48.515733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:48.515693 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.516022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:48.515990 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 03:09:48.545274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:48.539936 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.545274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:48.540004 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:09:49.443698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:49.443661 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.444401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:49.444368 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:49.526275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:49.525777 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.526275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:49.526027 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 03:09:49.542769 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:49.542721 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.543052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:49.543027 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:09:50.508283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:50.506197 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.575316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:50.532239 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.575316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:50.532330 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 03:09:50.575316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:50.541606 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.575316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:50.541680 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:09:50.575689 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:50.575320 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:51.433357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.433325 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:51.435058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.435032 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.435304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.435279 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:51.476236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.476207 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:51.526050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.526017 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.526727 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.526695 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 03:09:51.547384 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.547356 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.547948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.547591 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:09:51.872488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:51.872456 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:09:52.439277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.433894 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.439277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.433957 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:52.522275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.521701 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.522275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.521768 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 03:09:52.539996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.539966 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.540560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.540537 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 03:09:52.576147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.576110 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw"] Oct 31 03:09:52.582118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.582093 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.595339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.595298 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"distinct-w0-edb9e6-kserve-self-signed-certs\"" Oct 31 03:09:52.615553 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.615527 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw"] Oct 31 03:09:52.689267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.689207 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.689504 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.689449 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wzfb\" (UniqueName: \"kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.689640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.689619 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.689703 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.689655 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.689773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.689752 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.689835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.689811 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.796342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.793411 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wzfb\" (UniqueName: \"kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.796342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.793576 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.796342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.793610 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.796342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.793658 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.796342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.793715 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.796342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.793761 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.822284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.809432 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.822284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.810039 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.822284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.810765 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.835281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.833448 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.843413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.843382 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wzfb\" (UniqueName: \"kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.845411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.844276 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm\") pod \"distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:52.948946 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:52.948864 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:53.258183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.258151 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw"] Oct 31 03:09:53.260451 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:09:53.260360 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82cafdf5_57b6_46bd_9dd0_260613f4ccf2.slice/crio-9c19311992cb515f378d7466189bf1cd39851e2b6e5abe30ea0e85c5d65011c1 WatchSource:0}: Error finding container 9c19311992cb515f378d7466189bf1cd39851e2b6e5abe30ea0e85c5d65011c1: Status 404 returned error can't find the container with id 9c19311992cb515f378d7466189bf1cd39851e2b6e5abe30ea0e85c5d65011c1 Oct 31 03:09:53.430521 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.430426 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:09:53.430521 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.430500 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 03:09:53.703116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.703025 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" event={"ID":"82cafdf5-57b6-46bd-9dd0-260613f4ccf2","Type":"ContainerStarted","Data":"575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03"} Oct 31 03:09:53.703928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.703900 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" event={"ID":"82cafdf5-57b6-46bd-9dd0-260613f4ccf2","Type":"ContainerStarted","Data":"9c19311992cb515f378d7466189bf1cd39851e2b6e5abe30ea0e85c5d65011c1"} Oct 31 03:09:53.742219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.742147 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podStartSLOduration=1.74212701 podStartE2EDuration="1.74212701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:53.730239039 +0000 UTC m=+877.427406344" watchObservedRunningTime="2026-10-31 03:09:53.74212701 +0000 UTC m=+877.439294314" Oct 31 03:09:53.959324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:53.949876 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:09:54.092990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.092955 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:54.094598 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.094574 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:54.113530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.113498 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:09:54.264657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.264616 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:54.267013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.266638 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:54.269992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.269964 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:09:54.435024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.434994 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:54.435299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.435281 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:09:54.436133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:09:54.436115 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:10:02.973895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:02.973844 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:10:03.585975 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:03.585341 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:03.628274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:03.624331 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:03.895328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:03.894369 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:10:03.976398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:03.976356 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:04.020370 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:04.020334 4854 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:04.095768 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:04.095543 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:06.594713 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.594674 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2"] Oct 31 03:10:06.609227 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.609199 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.623061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.623034 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2"] Oct 31 03:10:06.634020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.633553 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:06.634479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.634459 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:06.695531 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.695504 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.695694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.695559 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.695694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.695682 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rln5z\" (UniqueName: \"kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.800899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.798190 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rln5z\" (UniqueName: \"kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.800899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.798467 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.800899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.798537 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.800899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.799550 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.805466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.805410 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.857466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.857386 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rln5z\" (UniqueName: \"kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z\") pod \"maas-api-revoke-sub-efc90a75-576f2\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:06.926592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:06.926566 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:07.208522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:07.205649 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2"] Oct 31 03:10:08.001264 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:08.001220 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" event={"ID":"7c42098c-af11-4461-a9ff-a84acc41add0","Type":"ContainerStarted","Data":"587b89e072f02fed9d2638bc6e77cc514b21ad1145cdc906d126f5b7fb1128ea"} Oct 31 03:10:08.001739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:08.001291 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" event={"ID":"7c42098c-af11-4461-a9ff-a84acc41add0","Type":"ContainerStarted","Data":"73dccc51c7600fd427c9a469994fa9c3082b77df595838f29a96881494d49028"} Oct 31 03:10:08.020481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:08.020435 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" podStartSLOduration=2.020422611 podStartE2EDuration="2.020422611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:08.017742981 +0000 UTC m=+891.714910285" watchObservedRunningTime="2026-10-31 03:10:08.020422611 +0000 UTC m=+891.717589916" Oct 31 03:10:09.009010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:09.008971 4854 generic.go:356] "Generic (PLEG): container finished" podID="7c42098c-af11-4461-a9ff-a84acc41add0" containerID="587b89e072f02fed9d2638bc6e77cc514b21ad1145cdc906d126f5b7fb1128ea" exitCode=0 Oct 31 03:10:09.009419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:09.009025 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" event={"ID":"7c42098c-af11-4461-a9ff-a84acc41add0","Type":"ContainerDied","Data":"587b89e072f02fed9d2638bc6e77cc514b21ad1145cdc906d126f5b7fb1128ea"} Oct 31 03:10:09.252206 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:10:09.252170 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:10:10.177815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.177794 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:10.280337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.263063 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca\") pod \"7c42098c-af11-4461-a9ff-a84acc41add0\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " Oct 31 03:10:10.280337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.268885 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca" pod "7c42098c-af11-4461-a9ff-a84acc41add0" (UID: "7c42098c-af11-4461-a9ff-a84acc41add0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:10.290743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.288790 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca" pod "7c42098c-af11-4461-a9ff-a84acc41add0" (UID: "7c42098c-af11-4461-a9ff-a84acc41add0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:10.292452 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.263379 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca\") pod \"7c42098c-af11-4461-a9ff-a84acc41add0\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " Oct 31 03:10:10.293667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.293639 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-xks-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:10.294188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.294167 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c42098c-af11-4461-a9ff-a84acc41add0-maas-api-openshift-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:10.394916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.394886 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z\" (UniqueName: \"kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z\") pod \"7c42098c-af11-4461-a9ff-a84acc41add0\" (UID: \"7c42098c-af11-4461-a9ff-a84acc41add0\") " Oct 31 03:10:10.397121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.397094 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z" pod "7c42098c-af11-4461-a9ff-a84acc41add0" (UID: "7c42098c-af11-4461-a9ff-a84acc41add0"). InnerVolumeSpecName "kube-api-access-rln5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:10.496258 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:10.496231 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rln5z\" (UniqueName: \"kubernetes.io/projected/7c42098c-af11-4461-a9ff-a84acc41add0-kube-api-access-rln5z\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:11.024293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:11.024234 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" event={"ID":"7c42098c-af11-4461-a9ff-a84acc41add0","Type":"ContainerDied","Data":"73dccc51c7600fd427c9a469994fa9c3082b77df595838f29a96881494d49028"} Oct 31 03:10:11.024599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:11.024565 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73dccc51c7600fd427c9a469994fa9c3082b77df595838f29a96881494d49028" Oct 31 03:10:11.024801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:11.024786 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2" Oct 31 03:10:12.961376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:12.961348 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:10:12.973478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:12.973455 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:10:13.361653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.361620 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb"] Oct 31 03:10:13.362236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.362222 4854 cpu_manager.go:414] "containerMap: removing container" podUID="7c42098c-af11-4461-a9ff-a84acc41add0" containerName="revoke-keys" Oct 31 03:10:13.362323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.362237 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c42098c-af11-4461-a9ff-a84acc41add0" containerName="revoke-keys" Oct 31 03:10:13.362397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.362387 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c42098c-af11-4461-a9ff-a84acc41add0" containerName="revoke-keys" Oct 31 03:10:13.366270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.366239 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.380685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.380660 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb"] Oct 31 03:10:13.490277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.489991 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.490277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.490171 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.490277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.490238 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.490604 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.490303 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.508304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.508216 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.508495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.508360 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.508495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.508390 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ftr\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.508495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.508446 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.508495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.508469 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.530436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.530406 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b"] Oct 31 03:10:13.534808 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.534786 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.550632 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.550610 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b"] Oct 31 03:10:13.609346 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609305 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609392 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609447 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609504 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609555 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609578 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ftr\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609657 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609682 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609797 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.609927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.609856 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.610296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.610237 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.610416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.610365 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.610584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.610560 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.610689 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.610670 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.612602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.612539 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.612722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.612701 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.620945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.620919 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ftr\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.620945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.620940 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.684510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.684474 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:13.711644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711614 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.711873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711781 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.711873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711806 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.711873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711837 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.711873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711883 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.712186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711939 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlrn\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.712186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.711984 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.712186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.712011 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.712186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.712106 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814048 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814104 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814132 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814183 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814207 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814242 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlrn\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.814324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814320 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.815456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814349 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.815456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814402 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.815456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.814554 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.815605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.815466 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.815605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.815489 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.815832 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.815805 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.816329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.816068 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.818714 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.818688 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.818806 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.818691 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.826480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.826453 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.827586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.827563 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlrn\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.849438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.849418 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:13.967609 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.967576 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9"] Oct 31 03:10:13.973738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.973646 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:13.981151 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:10:13.980435 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9e872_56dc_40ea_b858_f7212cc16ddc.slice/crio-e3855c18599b24f88cc7623276ae179f8657ffac0217c84d3ece8a3ed28875b7 WatchSource:0}: Error finding container e3855c18599b24f88cc7623276ae179f8657ffac0217c84d3ece8a3ed28875b7: Status 404 returned error can't find the container with id e3855c18599b24f88cc7623276ae179f8657ffac0217c84d3ece8a3ed28875b7 Oct 31 03:10:13.981151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.980569 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"distinct-2-w2-85cb59-kserve-self-signed-certs\"" Oct 31 03:10:13.984349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.984316 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:13.984439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.984396 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:13.984484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.984442 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:13.987334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.987145 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb"] Oct 31 03:10:13.993847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:13.991857 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9"] Oct 31 03:10:14.048859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.048825 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" event={"ID":"36f9e872-56dc-40ea-b858-f7212cc16ddc","Type":"ContainerStarted","Data":"e3855c18599b24f88cc7623276ae179f8657ffac0217c84d3ece8a3ed28875b7"} Oct 31 03:10:14.140322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.137705 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.140322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.137778 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.140322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.137811 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.140322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.137873 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8cj\" (UniqueName: \"kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.140322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.138097 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.140322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.138203 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.243178 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.243386 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.243432 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.243456 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.243498 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8cj\" (UniqueName: \"kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.243593 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.245638 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.246818 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.251287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.247130 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.270996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.263581 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b"] Oct 31 03:10:14.283318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.281169 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:14.283318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.281297 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:14.283318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.281345 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:14.293698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.293608 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.299342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.296672 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8cj\" (UniqueName: \"kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.302692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.302661 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs\") pod \"distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.303491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.303083 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:14.607888 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:10:14.607861 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ce5a45_f5ec_4a49_a765_440318032ee5.slice/crio-91bba88a8300e7383ca5e2da3135fa8e4b81d35757ace71e6a4e430a1b3766ff WatchSource:0}: Error finding container 91bba88a8300e7383ca5e2da3135fa8e4b81d35757ace71e6a4e430a1b3766ff: Status 404 returned error can't find the container with id 91bba88a8300e7383ca5e2da3135fa8e4b81d35757ace71e6a4e430a1b3766ff Oct 31 03:10:14.608024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:14.608000 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9"] Oct 31 03:10:15.059892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.059848 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" event={"ID":"5887b115-7ccf-41e8-b37f-eb5c2a326907","Type":"ContainerStarted","Data":"499bf728150767fb640e00565fa151966a74a3f7b73fe8fd806fbc3a318f7f71"} Oct 31 03:10:15.060355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.059917 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" event={"ID":"5887b115-7ccf-41e8-b37f-eb5c2a326907","Type":"ContainerStarted","Data":"1a39f0cfa1b02f034e5b923614ac58427900b340af3c34b79b513fec1d980f92"} Oct 31 03:10:15.062207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.062176 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" event={"ID":"b2ce5a45-f5ec-4a49-a765-440318032ee5","Type":"ContainerStarted","Data":"f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf"} Oct 31 03:10:15.062366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.062222 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" event={"ID":"b2ce5a45-f5ec-4a49-a765-440318032ee5","Type":"ContainerStarted","Data":"91bba88a8300e7383ca5e2da3135fa8e4b81d35757ace71e6a4e430a1b3766ff"} Oct 31 03:10:15.064017 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.063993 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" event={"ID":"36f9e872-56dc-40ea-b858-f7212cc16ddc","Type":"ContainerStarted","Data":"1fd05f3d1e63c2c6501fc8271722b948623a1e0c0e55c6d00a35a96e51017568"} Oct 31 03:10:15.083472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.083427 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podStartSLOduration=2.083413727 podStartE2EDuration="2.083413727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:15.079725788 +0000 UTC m=+898.776893095" watchObservedRunningTime="2026-10-31 03:10:15.083413727 +0000 UTC m=+898.780581031" Oct 31 03:10:15.110180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.110096 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podStartSLOduration=2.110076287 podStartE2EDuration="2.110076287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:15.108149933 +0000 UTC m=+898.805317245" watchObservedRunningTime="2026-10-31 03:10:15.110076287 +0000 UTC m=+898.807243596" Oct 31 03:10:15.313322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.304344 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:15.685174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.685098 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:15.686969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.686720 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:10:15.686969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.686768 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:10:15.850928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.850895 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:15.852326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.852305 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:10:15.852545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:15.852524 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 03:10:16.585616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:16.584313 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:10:16.585616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:16.584755 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:10:16.685365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:16.685330 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:10:16.685527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:16.685386 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:10:16.850155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:16.850072 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:10:16.850351 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:16.850149 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 03:10:17.685605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:17.685571 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:10:17.685986 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:17.685620 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:10:17.849975 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:17.849937 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:10:17.850156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:17.849998 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 03:10:18.685060 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:18.685026 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:10:18.685296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:18.685081 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:10:18.849885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:18.849850 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:10:18.850300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:18.849903 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 03:10:19.684851 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:19.684811 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:10:19.685026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:19.684879 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:10:19.850066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:19.850032 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:10:19.850466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:19.850083 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 03:10:20.685052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:20.685023 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:10:20.685231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:20.685076 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 03:10:20.850584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:20.850546 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:10:20.850989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:20.850614 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 03:10:21.690785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:21.690749 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:21.691766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:21.691739 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:21.692496 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:21.692477 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:10:21.712992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:21.712936 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podStartSLOduration=8.712922053 podStartE2EDuration="8.712922053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:15.137941647 +0000 UTC m=+898.835108952" watchObservedRunningTime="2026-10-31 03:10:21.712922053 +0000 UTC m=+905.410089357" Oct 31 03:10:22.221526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:22.221500 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:22.222345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:22.222319 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:22.223482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:22.223424 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:10:24.304022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:24.303988 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:24.316542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:24.316515 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:25.271699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:25.271670 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:27.875719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:27.875677 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2"] Oct 31 03:10:27.881691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:27.881667 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:27.898028 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:27.897998 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2"] Oct 31 03:10:28.043385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043351 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.043573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043402 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.043573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043519 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.043666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043572 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfpn\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.043722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043697 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.043870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043851 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.043927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043912 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.044018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.043997 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.044066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.044034 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.144992 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145078 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145110 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145141 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145171 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145215 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145280 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145331 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145371 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfpn\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145441 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145752 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.145860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145842 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.146080 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145891 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.146080 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.145906 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.147618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.147597 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.147852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.147833 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.161561 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.161511 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.164564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.164541 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfpn\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.196555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.196530 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp"] Oct 31 03:10:28.202583 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.202565 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.224434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.224407 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp"] Oct 31 03:10:28.231558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.231537 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:28.349229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349193 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349263 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349328 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349371 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349407 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbd2\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349467 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349512 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349618 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.349983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.349865 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.450779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450691 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.450779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450745 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450809 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450850 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450886 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbd2\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450942 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.450984 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451076 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451113 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451139 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451152 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451508 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451489 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451756 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451730 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.451934 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.451892 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.453918 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.453895 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.454232 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.454214 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.463388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.463362 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.463388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.463373 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbd2\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.522395 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.522369 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:28.527691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.527672 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2"] Oct 31 03:10:28.528947 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:10:28.528927 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01173b7_af83_4b2d_9970_80534aad8f63.slice/crio-64f51ba5e9135932ceda1cc6ac6baeb6dee3148516280b672dd1f7bec6a40cc5 WatchSource:0}: Error finding container 64f51ba5e9135932ceda1cc6ac6baeb6dee3148516280b672dd1f7bec6a40cc5: Status 404 returned error can't find the container with id 64f51ba5e9135932ceda1cc6ac6baeb6dee3148516280b672dd1f7bec6a40cc5 Oct 31 03:10:28.535047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.535011 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:28.535162 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.535095 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:28.535162 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.535138 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:28.933344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:28.922726 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp"] Oct 31 03:10:28.967796 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:10:28.955686 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6897173b_527e_4104_8248_eb37e8f3b3c0.slice/crio-8902f09dba7116621413e6b0ff748e2f8952026cd8a3ce95a5cb31c5e3f8c3eb WatchSource:0}: Error finding container 8902f09dba7116621413e6b0ff748e2f8952026cd8a3ce95a5cb31c5e3f8c3eb: Status 404 returned error can't find the container with id 8902f09dba7116621413e6b0ff748e2f8952026cd8a3ce95a5cb31c5e3f8c3eb Oct 31 03:10:29.022286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.016421 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:29.022286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.016516 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:29.022286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.016558 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:29.334276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.322782 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" event={"ID":"6897173b-527e-4104-8248-eb37e8f3b3c0","Type":"ContainerStarted","Data":"8902f09dba7116621413e6b0ff748e2f8952026cd8a3ce95a5cb31c5e3f8c3eb"} Oct 31 03:10:29.334276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.325763 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" event={"ID":"b01173b7-af83-4b2d-9970-80534aad8f63","Type":"ContainerStarted","Data":"c8be71f06e33fdb71d56f1d618acbd60eaa31abefef4f486f12f1413d6b23a32"} Oct 31 03:10:29.334276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.325817 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" event={"ID":"b01173b7-af83-4b2d-9970-80534aad8f63","Type":"ContainerStarted","Data":"64f51ba5e9135932ceda1cc6ac6baeb6dee3148516280b672dd1f7bec6a40cc5"} Oct 31 03:10:29.893286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.892329 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:29.978332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:29.974871 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:10:30.120426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.120379 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.235152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.235125 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:30.239612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.239575 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:30.239809 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.239783 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:30.320410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.320363 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.366278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.364347 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:10:30.431541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.431502 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" event={"ID":"6897173b-527e-4104-8248-eb37e8f3b3c0","Type":"ContainerStarted","Data":"b53514ece8e6707029b8e60d57649085b7e606f4cac910fed9ad6ee99af5e20f"} Oct 31 03:10:30.478274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.477270 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podStartSLOduration=3.477228685 podStartE2EDuration="3.477228685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:29.364000225 +0000 UTC m=+913.061167530" watchObservedRunningTime="2026-10-31 03:10:30.477228685 +0000 UTC m=+914.174395993" Oct 31 03:10:30.517331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.515647 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podStartSLOduration=2.5156276220000002 podStartE2EDuration="2.515627622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:30.477007029 +0000 UTC m=+914.174174335" watchObservedRunningTime="2026-10-31 03:10:30.515627622 +0000 UTC m=+914.212794927" Oct 31 03:10:30.527591 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.527563 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:30.547313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.545420 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:30.547313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.545482 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:30.895641 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.894332 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.895641 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:30.895090 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:10:31.195335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.193157 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.325286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.263008 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:10:31.339322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.334934 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:10:31.447366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.378673 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:31.447366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.378733 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:31.447366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.402882 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.447366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.403176 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:10:31.631284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.629051 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:31.631284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.629112 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:31.631284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.629529 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.647279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.631658 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:10:31.916474 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.916423 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:10:31.987557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.987515 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:10:31.988980 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:31.987803 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.232916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.232823 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:10:32.275956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.275922 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:32.276946 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.276910 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:32.335407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.335359 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:10:32.335996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.335963 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.340271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.336444 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:32.421273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.412150 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.545299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.501459 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 31 03:10:32.639447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.577353 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:32.639700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.639663 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:32.639846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.635404 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:10:32.640306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.635369 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:10:32.910342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.901958 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:10:32.930233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.930208 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:10:32.962443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.917942 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.995884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.995300 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.996156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.995343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:10:32.997738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.997718 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:10:32.997912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:32.997900 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:10:33.295782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.283327 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:10:33.322326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.322294 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:33.341607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.341567 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:33.341907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.341881 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:10:33.412654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.412610 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.527608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.527572 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:33.527871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.527841 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:33.557712 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.557630 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:10:33.646410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.646345 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:10:33.779062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.779032 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:10:33.814328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.785967 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:10:33.852285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.850574 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:10:33.972002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.971955 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:10:33.973310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.973153 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:10:33.981263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.974728 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:10:33.982289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:33.981376 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:10:34.247125 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:34.247091 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:34.247529 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:34.247496 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:34.525367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:34.523633 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:34.525367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:34.523690 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:35.235390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:35.235352 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:35.235652 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:35.235623 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:35.606411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:35.606331 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:35.606934 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:35.606900 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:36.240180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:36.240137 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:36.240402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:36.240196 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:36.523466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:36.522748 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:36.523466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:36.522818 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:37.239770 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:37.239736 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:37.240487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:37.240457 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:37.547362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:37.542684 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:37.547362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:37.542759 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:38.232505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:38.232465 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:10:38.232666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:38.232536 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 03:10:38.537185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:38.537149 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:10:38.537912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:38.537879 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 03:10:39.248634 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:39.248603 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:39.250490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:39.250468 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:39.259914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:39.259886 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:10:39.264324 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:10:39.264234 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:10:39.565046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:39.564966 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:39.585275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:39.572919 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:39.602066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:39.599529 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:10:58.071449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.071368 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx"] Oct 31 03:10:58.077273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.077233 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.090501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.090473 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx"] Oct 31 03:10:58.218889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.218858 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219068 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.218926 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219068 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219003 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219068 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219043 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-222mg\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219129 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219211 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219285 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219305 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.219434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.219413 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.299107 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.299076 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs"] Oct 31 03:10:58.303990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.303966 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.320907 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.320931 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs"] Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.320952 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.320996 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-222mg\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321050 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321125 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321195 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321224 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321320 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321470 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.322300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.321971 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.323849 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.323638 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.325742 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.324241 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.325742 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.325572 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.325923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.325770 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.327693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.327672 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.328192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.328170 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.348145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.348114 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-222mg\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.355742 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.355691 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.411274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.394392 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423560 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423655 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423716 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423742 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423799 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423891 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.423928 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zg7\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.424022 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.432078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.424063 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527134 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527092 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527158 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527179 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527223 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527305 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527337 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zg7\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527412 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527443 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527512 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.527918 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.527902 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.528402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.528377 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.528863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.528836 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.529168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.529119 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.530766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.530735 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.530993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.530979 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.531697 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.531673 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.540295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.540048 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.540413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.540391 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zg7\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.626370 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.626227 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:10:58.773789 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:10:58.773756 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76360468_ef71_4e93_b649_6e936156ee06.slice/crio-5ed6576ce4f5cdc6aceb6a4545d66658fef72196b48a7d5f28834741aa2f2833 WatchSource:0}: Error finding container 5ed6576ce4f5cdc6aceb6a4545d66658fef72196b48a7d5f28834741aa2f2833: Status 404 returned error can't find the container with id 5ed6576ce4f5cdc6aceb6a4545d66658fef72196b48a7d5f28834741aa2f2833 Oct 31 03:10:58.776535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.776501 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:58.776645 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.776582 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:58.776645 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.776625 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:58.776831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.776806 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx"] Oct 31 03:10:58.930564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.930527 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:58.930692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.930610 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:58.930692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.930656 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:10:58.936796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:58.936767 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs"] Oct 31 03:10:59.815264 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:59.815199 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" event={"ID":"76360468-ef71-4e93-b649-6e936156ee06","Type":"ContainerStarted","Data":"3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f"} Oct 31 03:10:59.815899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:59.815871 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" event={"ID":"76360468-ef71-4e93-b649-6e936156ee06","Type":"ContainerStarted","Data":"5ed6576ce4f5cdc6aceb6a4545d66658fef72196b48a7d5f28834741aa2f2833"} Oct 31 03:10:59.848461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:59.848420 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" event={"ID":"728dec5f-bdf3-45ff-ab1c-133444da826c","Type":"ContainerStarted","Data":"43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179"} Oct 31 03:10:59.850787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:59.850759 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" event={"ID":"728dec5f-bdf3-45ff-ab1c-133444da826c","Type":"ContainerStarted","Data":"9f0d598505d407e4626e1d69ab322b62e3bbfb6e2c4be8740cb582435a0fe0f8"} Oct 31 03:10:59.959276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:10:59.947292 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podStartSLOduration=1.94666803 podStartE2EDuration="1.94666803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:59.883870806 +0000 UTC m=+943.581038125" watchObservedRunningTime="2026-10-31 03:10:59.94666803 +0000 UTC m=+943.643835336" Oct 31 03:11:00.004522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.004452 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podStartSLOduration=2.004429018 podStartE2EDuration="2.004429018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:59.938918623 +0000 UTC m=+943.636085927" watchObservedRunningTime="2026-10-31 03:11:00.004429018 +0000 UTC m=+943.701596323" Oct 31 03:11:00.289416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.289368 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:00.344519 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.344479 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:00.402490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.402455 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:11:00.406423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.406395 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:00.406681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.406653 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:00.562331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.558345 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:00.637282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.636554 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:11:00.637282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.636688 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:00.637282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.636735 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:00.761284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.760358 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 31 03:11:00.790289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.789361 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:00.852334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.851374 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:11:00.936366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.936323 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:00.982536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:00.982491 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.052773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.052729 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:01.080277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.070377 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.080478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.080329 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.322276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.299344 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.322276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.300139 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:11:01.445274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.426626 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:11:01.477276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.471812 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:01.477276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.471991 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:01.563945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.561377 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.563945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.562168 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:11:01.687333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.646391 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:01.687333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.646472 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:01.802186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.802135 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.822285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.804315 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:11:01.861578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.861528 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.882112 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.864195 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:11:02.018854 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:01.976175 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:02.051731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.051693 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:02.053740 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.053705 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.064767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.064741 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:11:02.065004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.064986 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:11:02.065186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.065172 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:11:02.089413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.089367 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.183313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.182853 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:11:02.323397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.301684 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.461277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.445991 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.488480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.458764 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.607926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.584434 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.607926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.606293 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:02.607926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.606350 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:02.816298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.815599 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:02.816298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.815662 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:02.888383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.824825 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.047371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:02.888642 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.056628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.021369 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.058476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.058424 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:11:03.064926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.064882 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.065958 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.065771 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.101079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.101027 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.307376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.307328 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.540390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.540342 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:11:03.705332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.704450 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:03.736513 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.712412 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:03.767237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.767192 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:03.912620 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.029060 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.088609 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.088690 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.092362 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.092760 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.095183 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.114973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.095796 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:04.148869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.122891 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:04.336795 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.309575 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.574387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.561075 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:04.574387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.561150 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:04.575501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.574770 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.742346 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.742305 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:04.742640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:04.742593 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:05.060408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.060345 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.086283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.063325 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.104542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.104498 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.130270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.108531 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:05.131198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.105054 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:05.135272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.105024 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.175274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.174887 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.330938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.330838 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.353566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.353519 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:05.405464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.405430 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:05.405765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.405727 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:05.610392 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.604278 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.627525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.627493 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:05.821356 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:05.821305 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:06.019456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.009758 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:11:06.071768 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.066545 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.071768 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.067137 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:06.134283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.132442 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.134283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.133294 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:06.134283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.133719 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:06.181269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.175921 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:06.375848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.364491 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.415814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.401951 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:06.416163 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.416128 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:06.506318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.496711 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:11:06.506318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.496769 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:11:06.506318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.496802 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:11:06.506318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.496832 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:11:06.506318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.496862 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:11:06.506318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.496892 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:11:06.553000 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.552966 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:11:06.555342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.555319 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:11:06.652847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.652764 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:06.653523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:06.653491 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:07.417027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:07.411317 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.442662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:07.442616 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:07.645178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:07.644738 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.645178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:07.644798 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:08.397631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:08.397596 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.422396 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:08.422347 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:08.631283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:08.630849 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.631283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:08.630914 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:09.286724 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:11:09.286689 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:11:09.415381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:09.415345 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.415664 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:09.415634 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:09.627628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:09.627538 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.627628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:09.627601 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:10.396711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:10.396676 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:11:10.396910 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:10.396728 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 03:11:10.641626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:10.641589 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:11:10.642308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:10.642239 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 03:11:11.632411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:11.632382 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:11:11.632781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:11.632759 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:11:11.633471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:11.633452 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:11:12.438428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:12.438386 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:12.438817 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:12.438459 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:12.464022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:12.463998 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:11:12.464535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:12.464514 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:11:12.465211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:12.465194 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:11:28.434010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.433951 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv"] Oct 31 03:11:28.440566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.440509 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.529718 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.529685 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv"] Oct 31 03:11:28.529908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.529761 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b"] Oct 31 03:11:28.534850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.534827 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b"] Oct 31 03:11:28.534990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.534911 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.550879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.550847 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.550970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.550921 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551008 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551082 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551144 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551132 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551172 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551209 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551590 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.551762 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.551642 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.653357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653329 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.653532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653411 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.653532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653486 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.653532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653515 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.653695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653544 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.653748 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653702 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.653748 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653740 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.653861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653744 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.653861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653839 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.653964 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653924 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.653964 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653958 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.653989 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.654062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654011 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654110 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654115 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.654284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654156 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.654341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654313 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654426 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45c57\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.654522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654506 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654550 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654587 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.654970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.654947 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.655101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.655083 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.656133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.656112 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.656589 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.656569 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.665431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.665398 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.665560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.665539 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.756133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756103 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756155 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756199 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756224 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756301 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45c57\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756548 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756380 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756548 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756509 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756548 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756543 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756688 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756642 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.756824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.756800 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.757129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.757066 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.757129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.757097 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.757129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.757128 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.757420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.757325 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.758836 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.758813 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.758958 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.758941 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.768062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.768034 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45c57\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.768261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.768227 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:28.797303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.797279 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:28.850063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:28.850028 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:29.082336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.082309 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv"] Oct 31 03:11:29.083127 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:29.083103 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d33873_bc9b_4ccc_bee4_7e50adf8e730.slice/crio-e9173327bcec2b262839f46327b7b94683582bb70246b15046baad5a0256f260 WatchSource:0}: Error finding container e9173327bcec2b262839f46327b7b94683582bb70246b15046baad5a0256f260: Status 404 returned error can't find the container with id e9173327bcec2b262839f46327b7b94683582bb70246b15046baad5a0256f260 Oct 31 03:11:29.085705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.085672 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:29.085787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.085738 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:29.085787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.085767 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:29.169917 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.169895 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b"] Oct 31 03:11:29.171697 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:29.171674 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa21fe4_1a2d_42b7_a3b8_da71071c2917.slice/crio-1a89a07fda946a92f78dd00372e0b6cb300fe8a084ef2b1efda8c0720c26f887 WatchSource:0}: Error finding container 1a89a07fda946a92f78dd00372e0b6cb300fe8a084ef2b1efda8c0720c26f887: Status 404 returned error can't find the container with id 1a89a07fda946a92f78dd00372e0b6cb300fe8a084ef2b1efda8c0720c26f887 Oct 31 03:11:29.174741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.174710 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:29.174831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.174771 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:29.174831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.174802 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:29.379311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.379206 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" event={"ID":"1fa21fe4-1a2d-42b7-a3b8-da71071c2917","Type":"ContainerStarted","Data":"c70cefbb4c307f0c6edd48863131ef696f4022da3c672bd2fe4d9b7ac6b1404f"} Oct 31 03:11:29.379311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.379308 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" event={"ID":"1fa21fe4-1a2d-42b7-a3b8-da71071c2917","Type":"ContainerStarted","Data":"1a89a07fda946a92f78dd00372e0b6cb300fe8a084ef2b1efda8c0720c26f887"} Oct 31 03:11:29.381314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.381286 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" event={"ID":"67d33873-bc9b-4ccc-bee4-7e50adf8e730","Type":"ContainerStarted","Data":"203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef"} Oct 31 03:11:29.381455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.381340 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" event={"ID":"67d33873-bc9b-4ccc-bee4-7e50adf8e730","Type":"ContainerStarted","Data":"e9173327bcec2b262839f46327b7b94683582bb70246b15046baad5a0256f260"} Oct 31 03:11:29.468276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.462408 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podStartSLOduration=1.462388969 podStartE2EDuration="1.462388969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:29.435393537 +0000 UTC m=+973.132560843" watchObservedRunningTime="2026-10-31 03:11:29.462388969 +0000 UTC m=+973.159556274" Oct 31 03:11:29.491272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.483846 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podStartSLOduration=1.483825056 podStartE2EDuration="1.483825056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:29.460517998 +0000 UTC m=+973.157685325" watchObservedRunningTime="2026-10-31 03:11:29.483825056 +0000 UTC m=+973.180992351" Oct 31 03:11:29.835273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.829725 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:29.894284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:29.872582 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:30.543274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.540357 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:30.594446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.594203 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:30.770399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.770348 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:30.805089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.805004 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:30.805359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.805327 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:30.858333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.856433 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:30.858333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.856493 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:30.940294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:30.938562 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.114462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.114359 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.115040 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.115010 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 31 03:11:31.115220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.115204 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:31.115598 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.115574 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:31.337393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.337343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.488812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.488773 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:11:31.550420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.550376 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:31.551775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.551751 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:11:31.626466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.626421 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.664437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.632687 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.664437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.633210 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:11:31.823334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.797685 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:31.841276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.837337 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:31.983272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.937108 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:31.983272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.973688 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:31.983272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.941372 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:11:31.983272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:31.974040 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:11:32.116435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.116337 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.150329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.143489 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:32.180397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.180332 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:32.321432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.185410 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:11:32.321800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.321772 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:11:32.322012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.321989 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:11:32.355455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.338467 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.355455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.338791 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:11:32.443557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.443456 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.489628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.489573 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.567176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.567120 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.651550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.651498 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.653680 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.653616 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.842692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.842655 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:32.842990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.842957 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:32.994871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.994735 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.000474 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:32.996376 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.004564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.004332 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:33.148458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.148353 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.169277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.165689 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.200367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.200333 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:11:33.227472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.227442 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:11:33.371280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.339588 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:33.415189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.413363 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.495435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.495387 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.590232 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.590186 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.683446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.683344 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.710274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.685359 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:33.801129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.800655 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.801129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.800718 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:33.943361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.923602 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.943361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.923659 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:33.976491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:33.976430 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 31 03:11:34.011295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.010482 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:11:34.182540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.182490 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.329801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.199912 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:34.459284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.453381 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.459284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.454361 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:34.529416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.529219 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.608660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.608514 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.685025 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.684969 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.727674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.727624 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.816063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.816024 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.827894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.827852 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:34.888209 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.881618 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.888209 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:34.881675 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:35.010819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.010770 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.310275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.302797 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h"] Oct 31 03:11:35.449277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.334898 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:35.449277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.366599 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.488275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.468817 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.602960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.470682 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:35.603556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.470708 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:35.603966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.488558 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:35.604155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.604117 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.604290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.561400 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.609882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.609848 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:35.635214 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.635181 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h"] Oct 31 03:11:35.651412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.651327 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.663915 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.663879 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8a8b17ff-openshift-default-dockercfg-sp226\"" Oct 31 03:11:35.708415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.708376 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.713477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.713442 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.742495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.742460 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.742843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.717543 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.755456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.736341 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.788291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.751489 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.788291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.775449 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.788291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.775646 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.788291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.775823 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hb7\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.788291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.775874 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.788291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.775916 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.818473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.811937 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:35.818473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.811999 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:35.912963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.912858 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.916396 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.916366 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.916729 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.916708 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.917042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.917023 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hb7\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.917198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.917181 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.917360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.917342 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.917619 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.917589 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:35.922165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.922032 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.030612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.921230 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.030887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.921565 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.031115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.944791 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.031402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.031379 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.077373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:35.919028 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.085274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.032323 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.089277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.032369 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.114811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.114770 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.129240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.033318 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.145611 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.145573 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.158084 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.158049 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hb7\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7\") pod \"e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.226460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.226371 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:36.239282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.235343 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:36.336276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.319468 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:36.336276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.319560 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.336276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.335360 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.354276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.336981 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:36.636979 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.636912 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.660276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.641885 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:36.664529 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.663336 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:36.665233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.663378 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.675242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.663407 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:11:36.769201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.769045 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.771357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.771315 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.803969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.803938 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:36.825279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.804173 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:36.861269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.861213 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:36.869379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.869338 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:36.990601 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.973462 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:36.990601 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:36.973528 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.116237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.112390 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.116237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.112453 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.131662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.130359 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:11:37.345283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.343356 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:37.581277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.579427 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.581277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.579499 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.716467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.716397 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.717002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.716506 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.742276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.721164 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.742276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.721569 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:37.742276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.721887 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:37.742276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.722147 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:37.742276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.724558 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:37.770276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.769897 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.791279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.782901 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.814283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.810590 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.814283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.810651 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:37.850366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.841451 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.850366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.841517 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.858695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.856503 4854 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.858695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.856568 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.858695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.856801 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.858695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.856825 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:37.974460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.974411 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.990147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:37.990108 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.006670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.006607 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.158817 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.158725 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.191277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.188368 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:38.191277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.188442 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.337698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.337664 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:38.337991 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.337963 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.366663 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.366615 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.722407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.722358 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.723324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.723292 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:11:38.723709 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.723686 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:38.727557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.727533 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:38.756387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.756343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:11:38.816424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.812065 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.876667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:38.824352 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.051109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.015859 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.051109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.015914 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:39.051109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.016000 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.051109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.016024 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:39.190286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.177336 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.375879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.370170 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:39.619477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.619320 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 31 03:11:39.739367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.733104 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.739367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.734243 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:39.739367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.734770 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:11:39.744496 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.744465 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:11:39.879542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.761671 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.891738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.891596 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:39.912181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.827783 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.912181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.851950 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.912181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.898604 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:39.912181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.857803 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.912181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.898676 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:39.950136 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.935354 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:11:39.966566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:39.966171 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h"] Oct 31 03:11:40.099669 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.099566 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:40.099985 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.099958 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:40.100193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.100158 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:40.180048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.179998 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.441225 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.435422 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.441225 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.435850 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:40.805447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.788527 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.805447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.788683 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.805447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.789709 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:40.805447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.756416 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:11:40.888963 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:11:40.884904 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:11:40.961179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.954493 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:40.961179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.954551 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:40.973720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.971338 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:41.064103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.989470 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.064103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:40.989526 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:41.064103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.017635 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:11:41.495297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.495027 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:41.548338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.526062 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:11:41.616118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.561921 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:11:41.616118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.566442 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 31 03:11:41.745542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.745482 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.789065 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.818502 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:11:41.855549 4854 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.39s" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.855593 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.855640 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.855758 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.855815 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:11:41.856336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.855866 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:11:41.859174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.859110 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.859331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.859166 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:41.862763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.862717 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" event={"ID":"115985c1-3644-4971-b33f-789bcb827ad3","Type":"ContainerStarted","Data":"0f0b79b95b9ad77c5b2b37e6cb16dbf4b104838ecf3d06f7eb2d868f3b8e3fc8"} Oct 31 03:11:41.867888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.867869 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:11:41.871868 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.871843 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:11:41.875076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.875009 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:11:41.880747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.880726 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:11:41.968859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.968823 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:11:41.975307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:41.975219 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:11:42.896275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:42.888504 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:42.896275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:42.888602 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:42.929284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:42.920853 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:42.929284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:42.920923 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:42.967721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:42.967686 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" event={"ID":"115985c1-3644-4971-b33f-789bcb827ad3","Type":"ContainerStarted","Data":"079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18"} Oct 31 03:11:43.035292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.005212 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" podStartSLOduration=8.005193197 podStartE2EDuration="8.005193197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:43.002679137 +0000 UTC m=+986.699846441" watchObservedRunningTime="2026-10-31 03:11:43.005193197 +0000 UTC m=+986.702360508" Oct 31 03:11:43.344318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.344271 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:43.355575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.355531 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:43.811285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.807398 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:43.811285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.808597 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:43.811285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.808647 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:43.850334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.850289 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:43.850811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.850788 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:43.851018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:43.850995 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:44.115320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.114856 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:44.134275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.122693 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:11:44.773678 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.773645 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s"] Oct 31 03:11:44.780652 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.780382 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:44.783625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.783595 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s"] Oct 31 03:11:44.785343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.785323 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2ac00dc5-openshift-default-dockercfg-g2f54\"" Oct 31 03:11:44.857314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.833383 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:44.857314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.833442 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:44.859426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.857963 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:44.859426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.858005 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981305 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981410 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981466 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjbn\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981495 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981584 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981775 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981834 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981859 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.003276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:44.981949 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.091993 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092130 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092407 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092465 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092489 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092579 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092641 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092724 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.096907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.092774 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjbn\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.104907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.104866 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.106877 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.106698 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.111870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.111845 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.112199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.111799 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.117683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.117659 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.153311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.138393 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.169951 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.169916 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.186303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.184529 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjbn\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.205294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.203668 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy\") pod \"e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.475044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.474956 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:45.840572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.814870 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:45.840572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.814929 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:45.870101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.864533 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:45.870101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.864591 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:45.871732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.871709 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s"] Oct 31 03:11:45.874432 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:45.874400 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b785ab8_b765_4ec1_9dc1_ec75dbca43aa.slice/crio-fb4ecee1fa6129e2c14c9b46c5c0f808c4392e66b514cf18768a67e2f986e5c0 WatchSource:0}: Error finding container fb4ecee1fa6129e2c14c9b46c5c0f808c4392e66b514cf18768a67e2f986e5c0: Status 404 returned error can't find the container with id fb4ecee1fa6129e2c14c9b46c5c0f808c4392e66b514cf18768a67e2f986e5c0 Oct 31 03:11:45.877729 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.877699 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:45.877814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.877768 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:45.877814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:45.877795 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:11:46.198461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.198368 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" event={"ID":"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa","Type":"ContainerStarted","Data":"94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800"} Oct 31 03:11:46.198461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.198442 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" event={"ID":"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa","Type":"ContainerStarted","Data":"fb4ecee1fa6129e2c14c9b46c5c0f808c4392e66b514cf18768a67e2f986e5c0"} Oct 31 03:11:46.239317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.228628 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" podStartSLOduration=2.228608904 podStartE2EDuration="2.228608904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:46.221207493 +0000 UTC m=+989.918374799" watchObservedRunningTime="2026-10-31 03:11:46.228608904 +0000 UTC m=+989.925776208" Oct 31 03:11:46.499747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.477493 4854 patch_prober.go:29] interesting pod/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:11:46.499747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.477547 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" podUID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 03:11:46.548141 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.548107 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:46.799339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.798681 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:11:46.799339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.798753 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 03:11:46.853882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.853846 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:11:46.854210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:46.854164 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:11:47.481016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:47.480987 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:47.802841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:47.802809 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:47.803883 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:47.803861 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:11:47.856121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:47.856096 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:47.857501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:47.857477 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:11:48.294320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:48.294286 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:48.295204 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:48.295182 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:11:49.085706 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.085679 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56"] Oct 31 03:11:49.090415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.090397 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.096324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.096289 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2ac00dc5\"" Oct 31 03:11:49.096470 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.096444 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2ac00dc5-dockercfg-blnt2\"" Oct 31 03:11:49.100850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.100828 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56"] Oct 31 03:11:49.126390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.126360 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx"] Oct 31 03:11:49.130711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.130689 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.140305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.139428 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx"] Oct 31 03:11:49.283274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.279433 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.283274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.279544 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmfv\" (UniqueName: \"kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv\") pod \"payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.283274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.279687 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.283274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.279783 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv\") pod \"payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.397328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.383418 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.397328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.383514 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmfv\" (UniqueName: \"kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv\") pod \"payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.397328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.383630 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.397328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.383723 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv\") pod \"payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.397328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.385343 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.408275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.398243 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.444275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.438064 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmfv\" (UniqueName: \"kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv\") pod \"payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:49.444275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.442103 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv\") pod \"payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.457205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.453536 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:49.713331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:49.712556 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:50.024760 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:50.024725 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10a524c_8cab_4b5e_a354_c02be3852a3a.slice/crio-068e21729dd3c971be214c671cfaa55d1d74e08f36573e21fc3060e9ddcb3041 WatchSource:0}: Error finding container 068e21729dd3c971be214c671cfaa55d1d74e08f36573e21fc3060e9ddcb3041: Status 404 returned error can't find the container with id 068e21729dd3c971be214c671cfaa55d1d74e08f36573e21fc3060e9ddcb3041 Oct 31 03:11:50.024911 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.024795 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx"] Oct 31 03:11:50.051852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.051824 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56"] Oct 31 03:11:50.058698 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:50.058670 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5080e8b7_72c5_47c9_a5e2_3311656d64c3.slice/crio-89400a26c00a38caf5db0087dde1d2dcf8b096b941cf99bcab897607dabec4b9 WatchSource:0}: Error finding container 89400a26c00a38caf5db0087dde1d2dcf8b096b941cf99bcab897607dabec4b9: Status 404 returned error can't find the container with id 89400a26c00a38caf5db0087dde1d2dcf8b096b941cf99bcab897607dabec4b9 Oct 31 03:11:50.383015 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.382933 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" event={"ID":"a10a524c-8cab-4b5e-a354-c02be3852a3a","Type":"ContainerStarted","Data":"fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f"} Oct 31 03:11:50.383015 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.382993 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" event={"ID":"a10a524c-8cab-4b5e-a354-c02be3852a3a","Type":"ContainerStarted","Data":"068e21729dd3c971be214c671cfaa55d1d74e08f36573e21fc3060e9ddcb3041"} Oct 31 03:11:50.383576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.383028 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:11:50.384610 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.384584 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" event={"ID":"5080e8b7-72c5-47c9-a5e2-3311656d64c3","Type":"ContainerStarted","Data":"e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30"} Oct 31 03:11:50.384753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.384627 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" event={"ID":"5080e8b7-72c5-47c9-a5e2-3311656d64c3","Type":"ContainerStarted","Data":"89400a26c00a38caf5db0087dde1d2dcf8b096b941cf99bcab897607dabec4b9"} Oct 31 03:11:50.384753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.384676 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:11:50.410340 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.410301 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" podStartSLOduration=1.410289477 podStartE2EDuration="1.410289477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:50.405750241 +0000 UTC m=+994.102917544" watchObservedRunningTime="2026-10-31 03:11:50.410289477 +0000 UTC m=+994.107456780" Oct 31 03:11:50.431628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:50.431588 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" podStartSLOduration=1.431576711 podStartE2EDuration="1.431576711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:50.424113083 +0000 UTC m=+994.121280399" watchObservedRunningTime="2026-10-31 03:11:50.431576711 +0000 UTC m=+994.128744015" Oct 31 03:11:56.023740 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.023704 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747"] Oct 31 03:11:56.029420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.029399 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4"] Oct 31 03:11:56.029540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.029521 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.033683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.033661 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.038093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.038060 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1adda1ee\"" Oct 31 03:11:56.040941 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.040881 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747"] Oct 31 03:11:56.045197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.045175 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4"] Oct 31 03:11:56.134014 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.133980 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume\") pod \"payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.134202 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.134168 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.134267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.134223 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhc5z\" (UniqueName: \"kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z\") pod \"payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.134365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.134352 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6bf\" (UniqueName: \"kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf\") pod \"payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.264276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.239875 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhc5z\" (UniqueName: \"kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z\") pod \"payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.264276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.239982 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6bf\" (UniqueName: \"kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf\") pod \"payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.264276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.240134 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume\") pod \"payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.264276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.240649 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.264276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.241841 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.264276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.249404 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume\") pod \"payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.298545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.291400 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6bf\" (UniqueName: \"kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf\") pod \"payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.298545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.292040 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhc5z\" (UniqueName: \"kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z\") pod \"payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:56.357538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.357499 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:56.375580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:56.375544 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:57.084614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.084591 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747"] Oct 31 03:11:57.084963 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:57.084686 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacad21e_7ef5_4042_941b_f2aa7371fb51.slice/crio-8a764e90f8f104484dbf4278706a9ebbc7a75dcc8571206d89e7f4c178627ca6 WatchSource:0}: Error finding container 8a764e90f8f104484dbf4278706a9ebbc7a75dcc8571206d89e7f4c178627ca6: Status 404 returned error can't find the container with id 8a764e90f8f104484dbf4278706a9ebbc7a75dcc8571206d89e7f4c178627ca6 Oct 31 03:11:57.190395 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:11:57.190372 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b4a004_1a0e_44f3_9861_60e060e23dbf.slice/crio-04c913165c26f6040310d313005248f2b48334153fc716dab60ce67835cd84f5 WatchSource:0}: Error finding container 04c913165c26f6040310d313005248f2b48334153fc716dab60ce67835cd84f5: Status 404 returned error can't find the container with id 04c913165c26f6040310d313005248f2b48334153fc716dab60ce67835cd84f5 Oct 31 03:11:57.191437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.191416 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4"] Oct 31 03:11:57.338229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.338192 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8"] Oct 31 03:11:57.343934 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.343910 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.353197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.353132 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2ac00dc5-serving-cert\"" Oct 31 03:11:57.355285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.355228 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8"] Oct 31 03:11:57.422184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.422160 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.422374 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.422345 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.422438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.422391 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5nnc\" (UniqueName: \"kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.422528 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.422506 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.534276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.526346 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.534887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.534850 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5nnc\" (UniqueName: \"kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.543276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.535153 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.543725 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.543700 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.560230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.535169 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.560537 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.559272 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.575092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.574134 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.586167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.585778 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5nnc\" (UniqueName: \"kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc\") pod \"maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.597361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.591426 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" event={"ID":"eacad21e-7ef5-4042-941b-f2aa7371fb51","Type":"ContainerStarted","Data":"fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb"} Oct 31 03:11:57.597361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.591539 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" event={"ID":"eacad21e-7ef5-4042-941b-f2aa7371fb51","Type":"ContainerStarted","Data":"8a764e90f8f104484dbf4278706a9ebbc7a75dcc8571206d89e7f4c178627ca6"} Oct 31 03:11:57.597361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.593424 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:11:57.597361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.596717 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" event={"ID":"44b4a004-1a0e-44f3-9861-60e060e23dbf","Type":"ContainerStarted","Data":"84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa"} Oct 31 03:11:57.597361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.596774 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" event={"ID":"44b4a004-1a0e-44f3-9861-60e060e23dbf","Type":"ContainerStarted","Data":"04c913165c26f6040310d313005248f2b48334153fc716dab60ce67835cd84f5"} Oct 31 03:11:57.597361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.596958 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:11:57.638338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.631237 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" podStartSLOduration=1.631216565 podStartE2EDuration="1.631216565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:57.629581427 +0000 UTC m=+1001.326748731" watchObservedRunningTime="2026-10-31 03:11:57.631216565 +0000 UTC m=+1001.328383870" Oct 31 03:11:57.670582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.670549 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:11:57.707602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:57.707163 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" podStartSLOduration=1.707144135 podStartE2EDuration="1.707144135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:57.702981077 +0000 UTC m=+1001.400148407" watchObservedRunningTime="2026-10-31 03:11:57.707144135 +0000 UTC m=+1001.404311440" Oct 31 03:11:58.071477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:58.071451 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:11:58.089575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:58.089537 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8"] Oct 31 03:11:58.608391 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:58.608358 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" event={"ID":"625fbabc-a51e-4826-96b0-d34e2f06037f","Type":"ContainerStarted","Data":"ebb781f6e8495fb47f0b9214c9d3a98184ab11b46299b973ebd096d788537288"} Oct 31 03:11:59.667273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:59.665083 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" event={"ID":"625fbabc-a51e-4826-96b0-d34e2f06037f","Type":"ContainerStarted","Data":"a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c"} Oct 31 03:11:59.667273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:11:59.665184 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:12:01.442654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:01.442628 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:12:01.443049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:01.442695 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:12:01.504478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:01.504405 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" podStartSLOduration=3.9833439889999998 podStartE2EDuration="4.504382173s" totalImagesPullingTime="521.038184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:59.699095216 +0000 UTC m=+1003.396262521" watchObservedRunningTime="2026-10-31 03:12:01.504382173 +0000 UTC m=+1005.201549480" Oct 31 03:12:04.136221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.136182 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t"] Oct 31 03:12:04.142184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.142157 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.149399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.149373 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1adda1ee-serving-cert\"" Oct 31 03:12:04.156850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.156808 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t"] Oct 31 03:12:04.226343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.226313 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dncn\" (UniqueName: \"kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.226534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.226512 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.226610 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.226565 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.226737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.226718 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.329916 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.330099 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dncn\" (UniqueName: \"kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.330315 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.330373 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:04.331394 4854 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-serving-cert: secret "maas-api-e2e-derive-1adda1ee-serving-cert" not found Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:04.331468 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls podName:c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0 nodeName:}" failed. No retries permitted until 2026-10-31 03:12:04.831445544 +0000 UTC m=+1008.528612846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls") pod "maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" (UID: "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0") : secret "maas-api-e2e-derive-1adda1ee-serving-cert" not found Oct 31 03:12:04.333274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.332017 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.374928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.374810 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.380325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.376430 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dncn\" (UniqueName: \"kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.850788 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.843652 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:04.854294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:04.851213 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\") pod \"maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:05.083545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:05.076999 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:05.529594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:05.507871 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t"] Oct 31 03:12:05.670601 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:05.670572 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:12:05.704045 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:05.704012 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" event={"ID":"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0","Type":"ContainerStarted","Data":"e76e89d2165aa8f03a1215354b925bc0cb1a5502fbeb514eff1bfc6342b9768f"} Oct 31 03:12:06.715300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:06.715243 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" event={"ID":"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0","Type":"ContainerStarted","Data":"5f15462bef779cc7815bbfd60f735e47605d4db518e0978cbcd80b4b3a243c3e"} Oct 31 03:12:06.715751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:06.715340 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:06.735322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:06.735268 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" podStartSLOduration=2.313150659 podStartE2EDuration="2.735232091s" totalImagesPullingTime="422.081432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:06.733663627 +0000 UTC m=+1010.430830932" watchObservedRunningTime="2026-10-31 03:12:06.735232091 +0000 UTC m=+1010.432399397" Oct 31 03:12:08.612922 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:08.612894 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:12:09.502275 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:09.493920 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:12:09.715276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:09.709544 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:12:11.576274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:11.549563 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.577074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:11.577036 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.628458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:11.628414 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.641763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:11.641722 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:12:11.713199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:11.708432 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.738297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:11.736713 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.078037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.077994 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.156385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.156330 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx"] Oct 31 03:12:12.167108 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.167064 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.183175 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.182084 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:12:12.191291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.185450 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.250482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.250034 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:12:12.250482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.250119 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.362587 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.362750 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6tx\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.362888 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.362956 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.363007 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.363058 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.367320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.363146 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.491588 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.491668 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.491806 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6tx\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.491940 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.492002 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.492051 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.492737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.492099 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.579178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.577431 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.623057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.579736 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.649874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.649840 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.651379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.651354 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6tx\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.651575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.651551 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.682971 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.652863 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.693281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.652919 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.693281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.682381 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.716488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.715787 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.755286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.743349 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.793269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.793213 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:12:12.793532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.793512 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx"] Oct 31 03:12:12.815527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.813680 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:12:12.815527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.813749 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:12:12.815527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.813789 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:12:12.815527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.813835 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:12:12.815527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.813871 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:12:12.815527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.813913 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:12.873364 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.871325 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.918313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.889964 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:12.928372 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:12.928326 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:13.042294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.031590 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dmhxx\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:13.095301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.084331 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.095301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.085056 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:12:13.296047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.295752 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:13.297168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.296332 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.297168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.296851 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.297751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.297731 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:12:13.297950 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.297934 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:12:13.608322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.579444 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:12:13.684276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.683835 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.783283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.721338 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.783283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.721737 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:13.783283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.732354 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:12:13.783283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.748507 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.995848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:13.990988 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.103585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.100333 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.307602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.301762 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.368638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.327350 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:12:14.598443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.598343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.813403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.736367 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.813403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.736673 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.813403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.736865 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:12:14.813403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.757469 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.813403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:14.766332 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:12:15.325667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.325619 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.396184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.396130 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 03:12:15.684207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.610592 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.699414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.699348 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:12:15.781869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.781802 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:12:15.782878 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.782849 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:15.817312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.812401 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.817312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.813967 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.853791 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:15.853522 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.037317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:16.008383 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" podUID="9ca899b4-82ad-4c0f-9654-f9912c269a4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Oct 31 03:12:16.320277 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:16.286689 4854 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.825s" Oct 31 03:12:16.784336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:16.307093 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" podUID="c2b332bc-d96f-4af7-871e-be4a9a439e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.784336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:16.749978 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:12:16.784336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:16.750624 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:12:17.374275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.372832 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:12:17.388279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.387944 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:12:17.408414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.408353 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:17.514685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.467784 4854 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.020420762s: [/var/lib/containers/storage/overlay/1e4c7027cb410330c4617d72dbee5bbdeb2f006edf169ab8755b6cda82575b02/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5nsnk_dcf9a703-8b57-41ae-a801-cee51331a479/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:12:17.514880 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:16.373677 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:12:17.515023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.507454 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:12:17.555302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:16.376330 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:12:17.555302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.548060 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:12:17.585948 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:17.585054 4854 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.058s" Oct 31 03:12:17.585948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.585091 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" event={"ID":"6e17e36c-cf1f-485f-a02f-a2c57b1f699f","Type":"ContainerStarted","Data":"c0868c111a258e1285031d08155345f715b9b2ea7108d88b379135946875b247"} Oct 31 03:12:17.588676 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.588647 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:12:17.588808 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.588693 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx"] Oct 31 03:12:17.588808 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.588780 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:12:17.588916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.588829 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:12:17.588916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.588872 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:12:17.678909 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.676793 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:12:17.679495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.679475 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:12:17.681203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.681187 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:12:17.777359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.752285 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:12:17.784432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.759685 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:17.785425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.759275 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:12:17.812296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.787449 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:12:17.848803 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.848757 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:12:17.867273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:17.867010 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:12:18.280189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:18.280153 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" event={"ID":"6e17e36c-cf1f-485f-a02f-a2c57b1f699f","Type":"ContainerStarted","Data":"2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377"} Oct 31 03:12:18.281344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:18.281325 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:18.281736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:18.281714 4854 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dmhxx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:12:18.281834 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:18.281784 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:12:18.326446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:18.326381 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" podStartSLOduration=7.326360861 podStartE2EDuration="7.326360861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:18.30580537 +0000 UTC m=+1022.002972677" watchObservedRunningTime="2026-10-31 03:12:18.326360861 +0000 UTC m=+1022.023528166" Oct 31 03:12:19.298277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:19.296358 4854 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dmhxx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:12:19.298277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:19.296425 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:12:20.334390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:20.334354 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:12:24.805011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:24.804333 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx"] Oct 31 03:12:24.812434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:24.812406 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56"] Oct 31 03:12:27.515543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:27.515511 4854 generic.go:356] "Generic (PLEG): container finished" podID="115985c1-3644-4971-b33f-789bcb827ad3" containerID="079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18" exitCode=0 Oct 31 03:12:27.516017 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:27.515628 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" event={"ID":"115985c1-3644-4971-b33f-789bcb827ad3","Type":"ContainerDied","Data":"079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18"} Oct 31 03:12:27.516146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:27.516121 4854 scope.go:122] "RemoveContainer" containerID="079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18" Oct 31 03:12:27.517696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:27.517669 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:12:27.517773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:27.517737 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:12:27.517773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:27.517768 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:12:28.525760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:28.525726 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" event={"ID":"115985c1-3644-4971-b33f-789bcb827ad3","Type":"ContainerStarted","Data":"67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58"} Oct 31 03:12:29.114741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:29.114698 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:12:29.340179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:29.340148 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:12:29.341048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:29.341027 4854 patch_prober.go:29] interesting pod/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:12:29.341150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:29.341091 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 03:12:29.806361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:29.806318 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" podUID="a10a524c-8cab-4b5e-a354-c02be3852a3a" containerName="payload-processing" containerID="cri-o://fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f" gracePeriod=25 Oct 31 03:12:29.813451 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:29.813406 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" podUID="5080e8b7-72c5-47c9-a5e2-3311656d64c3" containerName="payload-pre-processing" containerID="cri-o://e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30" gracePeriod=25 Oct 31 03:12:30.185080 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.185053 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:12:30.325275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.316892 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:12:30.330283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.328161 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume\") pod \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " Oct 31 03:12:30.330283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.328280 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv\" (UniqueName: \"kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv\") pod \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\" (UID: \"5080e8b7-72c5-47c9-a5e2-3311656d64c3\") " Oct 31 03:12:30.330608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.330297 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume" pod "5080e8b7-72c5-47c9-a5e2-3311656d64c3" (UID: "5080e8b7-72c5-47c9-a5e2-3311656d64c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:30.369327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.366707 4854 patch_prober.go:29] interesting pod/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:12:30.369327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.366784 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 03:12:30.383274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.380174 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv" pod "5080e8b7-72c5-47c9-a5e2-3311656d64c3" (UID: "5080e8b7-72c5-47c9-a5e2-3311656d64c3"). InnerVolumeSpecName "kube-api-access-4tmfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:30.436275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.430838 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume\") pod \"a10a524c-8cab-4b5e-a354-c02be3852a3a\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " Oct 31 03:12:30.436275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.431118 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv\") pod \"a10a524c-8cab-4b5e-a354-c02be3852a3a\" (UID: \"a10a524c-8cab-4b5e-a354-c02be3852a3a\") " Oct 31 03:12:30.436275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.431929 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080e8b7-72c5-47c9-a5e2-3311656d64c3-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:30.436275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.431951 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmfv\" (UniqueName: \"kubernetes.io/projected/5080e8b7-72c5-47c9-a5e2-3311656d64c3-kube-api-access-4tmfv\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:30.452283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.437528 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume" pod "a10a524c-8cab-4b5e-a354-c02be3852a3a" (UID: "a10a524c-8cab-4b5e-a354-c02be3852a3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:30.455415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.455376 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv" pod "a10a524c-8cab-4b5e-a354-c02be3852a3a" (UID: "a10a524c-8cab-4b5e-a354-c02be3852a3a"). InnerVolumeSpecName "kube-api-access-mwzwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:30.533723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.533686 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/a10a524c-8cab-4b5e-a354-c02be3852a3a-kube-api-access-mwzwv\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:30.533881 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.533726 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10a524c-8cab-4b5e-a354-c02be3852a3a-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:30.568773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.568741 4854 generic.go:356] "Generic (PLEG): container finished" podID="a10a524c-8cab-4b5e-a354-c02be3852a3a" containerID="fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f" exitCode=0 Oct 31 03:12:30.568938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.568785 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" Oct 31 03:12:30.568938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.568826 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" event={"ID":"a10a524c-8cab-4b5e-a354-c02be3852a3a","Type":"ContainerDied","Data":"fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f"} Oct 31 03:12:30.568938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.568896 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx" event={"ID":"a10a524c-8cab-4b5e-a354-c02be3852a3a","Type":"ContainerDied","Data":"068e21729dd3c971be214c671cfaa55d1d74e08f36573e21fc3060e9ddcb3041"} Oct 31 03:12:30.568938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.568927 4854 scope.go:122] "RemoveContainer" containerID="fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f" Oct 31 03:12:30.570549 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.570525 4854 generic.go:356] "Generic (PLEG): container finished" podID="5080e8b7-72c5-47c9-a5e2-3311656d64c3" containerID="e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30" exitCode=0 Oct 31 03:12:30.570664 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.570583 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" Oct 31 03:12:30.570725 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.570610 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" event={"ID":"5080e8b7-72c5-47c9-a5e2-3311656d64c3","Type":"ContainerDied","Data":"e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30"} Oct 31 03:12:30.570725 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.570716 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56" event={"ID":"5080e8b7-72c5-47c9-a5e2-3311656d64c3","Type":"ContainerDied","Data":"89400a26c00a38caf5db0087dde1d2dcf8b096b941cf99bcab897607dabec4b9"} Oct 31 03:12:30.580450 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.580431 4854 scope.go:122] "RemoveContainer" containerID="fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f" Oct 31 03:12:30.580745 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:30.580726 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f\": container with ID starting with fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f not found: ID does not exist" containerID="fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f" Oct 31 03:12:30.580802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.580752 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f"} err="failed to get container status \"fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f\": rpc error: code = NotFound desc = could not find container \"fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f\": container with ID starting with fac09ba34400df45dc14f9ec9e205f7013ba030794fcfca61b96095a741b2a6f not found: ID does not exist" Oct 31 03:12:30.580802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.580769 4854 scope.go:122] "RemoveContainer" containerID="e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30" Oct 31 03:12:30.616801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.601365 4854 scope.go:122] "RemoveContainer" containerID="e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30" Oct 31 03:12:30.617070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.617016 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx"] Oct 31 03:12:30.618404 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:30.618371 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30\": container with ID starting with e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30 not found: ID does not exist" containerID="e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30" Oct 31 03:12:30.618554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.618411 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30"} err="failed to get container status \"e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30\": rpc error: code = NotFound desc = could not find container \"e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30\": container with ID starting with e9314b98c7dbebedf9067ac2fcd9cb2c8927ad166a94878c171908d897f5aa30 not found: ID does not exist" Oct 31 03:12:30.622767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.622717 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ac00dc5-7855c789dc-q59jx"] Oct 31 03:12:30.631364 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.631341 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56"] Oct 31 03:12:30.635217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:30.635196 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ac00dc5-5cb4cc4775-jbq56"] Oct 31 03:12:31.340137 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:31.340104 4854 patch_prober.go:29] interesting pod/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:12:31.340619 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:31.340162 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 03:12:32.360093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.359601 4854 patch_prober.go:29] interesting pod/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:12:32.360093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.359665 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 03:12:32.427723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.427691 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h"] Oct 31 03:12:32.437278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.428266 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" containerID="cri-o://67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58" gracePeriod=30 Oct 31 03:12:32.713357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.710043 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:12:32.759500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.759458 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:32.765789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.765734 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:32.782285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.782224 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5080e8b7-72c5-47c9-a5e2-3311656d64c3" path="/var/lib/kubelet/pods/5080e8b7-72c5-47c9-a5e2-3311656d64c3/volumes" Oct 31 03:12:32.782837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:32.782818 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10a524c-8cab-4b5e-a354-c02be3852a3a" path="/var/lib/kubelet/pods/a10a524c-8cab-4b5e-a354-c02be3852a3a/volumes" Oct 31 03:12:33.743860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:33.743822 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8"] Oct 31 03:12:33.744633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:33.744600 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" podUID="625fbabc-a51e-4826-96b0-d34e2f06037f" containerName="maas-api" containerID="cri-o://a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c" gracePeriod=30 Oct 31 03:12:34.667278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.666039 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:12:34.762466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.762437 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc\" (UniqueName: \"kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc\") pod \"625fbabc-a51e-4826-96b0-d34e2f06037f\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " Oct 31 03:12:34.762928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.762580 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs\") pod \"625fbabc-a51e-4826-96b0-d34e2f06037f\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " Oct 31 03:12:34.762928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.762670 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca\") pod \"625fbabc-a51e-4826-96b0-d34e2f06037f\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " Oct 31 03:12:34.762928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.762753 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls\") pod \"625fbabc-a51e-4826-96b0-d34e2f06037f\" (UID: \"625fbabc-a51e-4826-96b0-d34e2f06037f\") " Oct 31 03:12:34.763087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.763064 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca" pod "625fbabc-a51e-4826-96b0-d34e2f06037f" (UID: "625fbabc-a51e-4826-96b0-d34e2f06037f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:34.763599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.763574 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625fbabc-a51e-4826-96b0-d34e2f06037f-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:34.764942 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.764922 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs" pod "625fbabc-a51e-4826-96b0-d34e2f06037f" (UID: "625fbabc-a51e-4826-96b0-d34e2f06037f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:34.765357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.765342 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls" pod "625fbabc-a51e-4826-96b0-d34e2f06037f" (UID: "625fbabc-a51e-4826-96b0-d34e2f06037f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:34.765445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.765395 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc" pod "625fbabc-a51e-4826-96b0-d34e2f06037f" (UID: "625fbabc-a51e-4826-96b0-d34e2f06037f"). InnerVolumeSpecName "kube-api-access-j5nnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:34.788758 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.788732 4854 generic.go:356] "Generic (PLEG): container finished" podID="625fbabc-a51e-4826-96b0-d34e2f06037f" containerID="a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c" exitCode=0 Oct 31 03:12:34.788876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.788804 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" Oct 31 03:12:34.788876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.788815 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" event={"ID":"625fbabc-a51e-4826-96b0-d34e2f06037f","Type":"ContainerDied","Data":"a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c"} Oct 31 03:12:34.788876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.788872 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8" event={"ID":"625fbabc-a51e-4826-96b0-d34e2f06037f","Type":"ContainerDied","Data":"ebb781f6e8495fb47f0b9214c9d3a98184ab11b46299b973ebd096d788537288"} Oct 31 03:12:34.788981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.788901 4854 scope.go:122] "RemoveContainer" containerID="a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c" Oct 31 03:12:34.820415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.811966 4854 scope.go:122] "RemoveContainer" containerID="a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c" Oct 31 03:12:34.820415 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:34.812815 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c\": container with ID starting with a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c not found: ID does not exist" containerID="a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c" Oct 31 03:12:34.820415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.812859 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c"} err="failed to get container status \"a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c\": rpc error: code = NotFound desc = could not find container \"a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c\": container with ID starting with a6204fdb7dd890a54df9db19b35e4fabd9aea0303eaab02c9ceed1e99432af3c not found: ID does not exist" Oct 31 03:12:34.825637 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.825602 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8"] Oct 31 03:12:34.831349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.831326 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ac00dc5-774c59dbb7-thvf8"] Oct 31 03:12:34.865283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.865229 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:34.865283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.865284 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5nnc\" (UniqueName: \"kubernetes.io/projected/625fbabc-a51e-4826-96b0-d34e2f06037f-kube-api-access-j5nnc\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:34.865469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:34.865296 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625fbabc-a51e-4826-96b0-d34e2f06037f-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:35.237673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:35.237642 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4"] Oct 31 03:12:35.251612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:35.251582 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747"] Oct 31 03:12:36.008517 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:36.008479 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s"] Oct 31 03:12:36.009018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:36.008738 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" podUID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" containerName="istio-proxy" containerID="cri-o://94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800" gracePeriod=30 Oct 31 03:12:36.542300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:36.540725 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625fbabc-a51e-4826-96b0-d34e2f06037f" path="/var/lib/kubelet/pods/625fbabc-a51e-4826-96b0-d34e2f06037f/volumes" Oct 31 03:12:39.279571 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:39.279534 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:12:40.238721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.238666 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" podUID="44b4a004-1a0e-44f3-9861-60e060e23dbf" containerName="payload-processing" containerID="cri-o://84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa" gracePeriod=25 Oct 31 03:12:40.252741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.252708 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" podUID="eacad21e-7ef5-4042-941b-f2aa7371fb51" containerName="payload-pre-processing" containerID="cri-o://fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb" gracePeriod=25 Oct 31 03:12:40.408952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.408925 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:12:40.496896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.496871 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.496968 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497019 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497071 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497148 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497183 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497217 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497330 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497552 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs\") pod \"115985c1-3644-4971-b33f-789bcb827ad3\" (UID: \"115985c1-3644-4971-b33f-789bcb827ad3\") " Oct 31 03:12:40.497812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.497792 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:40.498105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.498083 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:40.498236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.498209 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:40.498456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.498433 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:40.500155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.498600 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:40.500155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.500069 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115985c1-3644-4971-b33f-789bcb827ad3-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.500155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.500089 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.500155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.500105 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.500155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.500117 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.500155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.500130 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.504568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.504532 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:12:40.504669 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.504615 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:40.504824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.504798 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "kube-api-access-b6hb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:40.506228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.506205 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t"] Oct 31 03:12:40.506494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.506451 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" podUID="c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" containerName="maas-api" containerID="cri-o://5f15462bef779cc7815bbfd60f735e47605d4db518e0978cbcd80b4b3a243c3e" gracePeriod=30 Oct 31 03:12:40.510036 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.510011 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token" pod "115985c1-3644-4971-b33f-789bcb827ad3" (UID: "115985c1-3644-4971-b33f-789bcb827ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:40.602173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.602086 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115985c1-3644-4971-b33f-789bcb827ad3-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.602173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.602114 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115985c1-3644-4971-b33f-789bcb827ad3-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.602173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.602128 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hb7\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-kube-api-access-b6hb7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.602173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.602143 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115985c1-3644-4971-b33f-789bcb827ad3-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.621800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.621772 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:12:40.626663 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.626640 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:12:40.704956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.704918 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume\") pod \"44b4a004-1a0e-44f3-9861-60e060e23dbf\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " Oct 31 03:12:40.705146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.705038 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z\" (UniqueName: \"kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z\") pod \"44b4a004-1a0e-44f3-9861-60e060e23dbf\" (UID: \"44b4a004-1a0e-44f3-9861-60e060e23dbf\") " Oct 31 03:12:40.705222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.705146 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf\" (UniqueName: \"kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf\") pod \"eacad21e-7ef5-4042-941b-f2aa7371fb51\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " Oct 31 03:12:40.705342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.705317 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume\") pod \"eacad21e-7ef5-4042-941b-f2aa7371fb51\" (UID: \"eacad21e-7ef5-4042-941b-f2aa7371fb51\") " Oct 31 03:12:40.706752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.706723 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume" pod "eacad21e-7ef5-4042-941b-f2aa7371fb51" (UID: "eacad21e-7ef5-4042-941b-f2aa7371fb51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:40.707165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.707144 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume" pod "44b4a004-1a0e-44f3-9861-60e060e23dbf" (UID: "44b4a004-1a0e-44f3-9861-60e060e23dbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:40.709485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.709461 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z" pod "44b4a004-1a0e-44f3-9861-60e060e23dbf" (UID: "44b4a004-1a0e-44f3-9861-60e060e23dbf"). InnerVolumeSpecName "kube-api-access-lhc5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:40.709780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.709754 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf" pod "eacad21e-7ef5-4042-941b-f2aa7371fb51" (UID: "eacad21e-7ef5-4042-941b-f2aa7371fb51"). InnerVolumeSpecName "kube-api-access-hp6bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:40.808152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.808123 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b4a004-1a0e-44f3-9861-60e060e23dbf-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.808152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.808149 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhc5z\" (UniqueName: \"kubernetes.io/projected/44b4a004-1a0e-44f3-9861-60e060e23dbf-kube-api-access-lhc5z\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.808152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.808160 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6bf\" (UniqueName: \"kubernetes.io/projected/eacad21e-7ef5-4042-941b-f2aa7371fb51-kube-api-access-hp6bf\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.808152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.808168 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eacad21e-7ef5-4042-941b-f2aa7371fb51-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:40.873480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.871517 4854 generic.go:356] "Generic (PLEG): container finished" podID="44b4a004-1a0e-44f3-9861-60e060e23dbf" containerID="84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa" exitCode=0 Oct 31 03:12:40.873480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.871693 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" event={"ID":"44b4a004-1a0e-44f3-9861-60e060e23dbf","Type":"ContainerDied","Data":"84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa"} Oct 31 03:12:40.873480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.871751 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" event={"ID":"44b4a004-1a0e-44f3-9861-60e060e23dbf","Type":"ContainerDied","Data":"04c913165c26f6040310d313005248f2b48334153fc716dab60ce67835cd84f5"} Oct 31 03:12:40.873480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.871782 4854 scope.go:122] "RemoveContainer" containerID="84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa" Oct 31 03:12:40.873480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.872025 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4" Oct 31 03:12:40.975281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.959017 4854 generic.go:356] "Generic (PLEG): container finished" podID="115985c1-3644-4971-b33f-789bcb827ad3" containerID="67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58" exitCode=0 Oct 31 03:12:40.975281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.959347 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" event={"ID":"115985c1-3644-4971-b33f-789bcb827ad3","Type":"ContainerDied","Data":"67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58"} Oct 31 03:12:40.975281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.959405 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" event={"ID":"115985c1-3644-4971-b33f-789bcb827ad3","Type":"ContainerDied","Data":"0f0b79b95b9ad77c5b2b37e6cb16dbf4b104838ecf3d06f7eb2d868f3b8e3fc8"} Oct 31 03:12:40.975281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:40.959510 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h" Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.026863 4854 scope.go:122] "RemoveContainer" containerID="84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa" Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.028029 4854 generic.go:356] "Generic (PLEG): container finished" podID="eacad21e-7ef5-4042-941b-f2aa7371fb51" containerID="fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb" exitCode=0 Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.028112 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" event={"ID":"eacad21e-7ef5-4042-941b-f2aa7371fb51","Type":"ContainerDied","Data":"fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb"} Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.028166 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" event={"ID":"eacad21e-7ef5-4042-941b-f2aa7371fb51","Type":"ContainerDied","Data":"8a764e90f8f104484dbf4278706a9ebbc7a75dcc8571206d89e7f4c178627ca6"} Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.037445 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:41.037801 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa\": container with ID starting with 84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa not found: ID does not exist" containerID="84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa" Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.037845 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa"} err="failed to get container status \"84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa\": rpc error: code = NotFound desc = could not find container \"84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa\": container with ID starting with 84edbe375c2abe98bf21939e90cce96d5f0e2e5273e9db185f0b04b5bcf67baa not found: ID does not exist" Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.037869 4854 scope.go:122] "RemoveContainer" containerID="67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58" Oct 31 03:12:41.047386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.038014 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747" Oct 31 03:12:41.056697 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.056667 4854 generic.go:356] "Generic (PLEG): container finished" podID="c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" containerID="5f15462bef779cc7815bbfd60f735e47605d4db518e0978cbcd80b4b3a243c3e" exitCode=0 Oct 31 03:12:41.056896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.056879 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" event={"ID":"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0","Type":"ContainerDied","Data":"5f15462bef779cc7815bbfd60f735e47605d4db518e0978cbcd80b4b3a243c3e"} Oct 31 03:12:41.129425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.127035 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs\") pod \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " Oct 31 03:12:41.129425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.127136 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca\") pod \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " Oct 31 03:12:41.129425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.127210 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\") pod \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " Oct 31 03:12:41.132087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.130332 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca" pod "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" (UID: "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:41.132087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.130431 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn\" (UniqueName: \"kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn\") pod \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\" (UID: \"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0\") " Oct 31 03:12:41.133731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.133707 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:41.138438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.138396 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs" pod "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" (UID: "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:41.139341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.139230 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn" pod "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" (UID: "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0"). InnerVolumeSpecName "kube-api-access-5dncn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:41.159275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.150546 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls" pod "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" (UID: "c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:41.159275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.153331 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4"] Oct 31 03:12:41.175583 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.163752 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1adda1ee-67c9497cff-t8zz4"] Oct 31 03:12:41.241773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.236899 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:41.241773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.236945 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:41.241773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.236963 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dncn\" (UniqueName: \"kubernetes.io/projected/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0-kube-api-access-5dncn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:41.266674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.266643 4854 scope.go:122] "RemoveContainer" containerID="079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18" Oct 31 03:12:41.272201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.272177 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747"] Oct 31 03:12:41.278691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.278665 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1adda1ee-6dbb887ff5-zw747"] Oct 31 03:12:41.287344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.287315 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h"] Oct 31 03:12:41.289031 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.289014 4854 scope.go:122] "RemoveContainer" containerID="67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58" Oct 31 03:12:41.289496 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:41.289471 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58\": container with ID starting with 67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58 not found: ID does not exist" containerID="67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58" Oct 31 03:12:41.289615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.289509 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58"} err="failed to get container status \"67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58\": rpc error: code = NotFound desc = could not find container \"67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58\": container with ID starting with 67022fbb717bd2837771f870115b6425d44fbb855552e87a3ddd1e3ed3462e58 not found: ID does not exist" Oct 31 03:12:41.289615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.289536 4854 scope.go:122] "RemoveContainer" containerID="079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18" Oct 31 03:12:41.290021 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:41.289965 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18\": container with ID starting with 079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18 not found: ID does not exist" containerID="079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18" Oct 31 03:12:41.290116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.290031 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18"} err="failed to get container status \"079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18\": rpc error: code = NotFound desc = could not find container \"079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18\": container with ID starting with 079d41de17d12f248a80e95e3ad62a8a267a0ff7f0a1860f38a7dec7fe9b1b18 not found: ID does not exist" Oct 31 03:12:41.290116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.290053 4854 scope.go:122] "RemoveContainer" containerID="fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb" Oct 31 03:12:41.294330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.293776 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8a8b17ff-openshift-default-56bbfcd875-kqz2h"] Oct 31 03:12:41.307922 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.307792 4854 scope.go:122] "RemoveContainer" containerID="fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb" Oct 31 03:12:41.309127 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:41.308758 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb\": container with ID starting with fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb not found: ID does not exist" containerID="fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb" Oct 31 03:12:41.309127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.308794 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb"} err="failed to get container status \"fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb\": rpc error: code = NotFound desc = could not find container \"fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb\": container with ID starting with fb964d1c7312f3ab75a0600c3c795825be805fd5dbd3bf14ebcf767382c224cb not found: ID does not exist" Oct 31 03:12:41.309127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.308819 4854 scope.go:122] "RemoveContainer" containerID="5f15462bef779cc7815bbfd60f735e47605d4db518e0978cbcd80b4b3a243c3e" Oct 31 03:12:41.772297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.771584 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930017 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930171 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930295 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930336 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930363 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930406 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930444 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930504 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.930593 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo\") pod \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\" (UID: \"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa\") " Oct 31 03:12:41.935772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.934842 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:12:41.945282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.945230 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:41.947383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.945540 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:41.947383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.945807 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:41.947383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.946240 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:41.947383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.946534 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:41.951457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.951433 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:41.951670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.951647 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "kube-api-access-tdjbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:41.953974 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:41.953936 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token" pod "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" (UID: "5b785ab8-b765-4ec1-9dc1-ec75dbca43aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:42.033633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033608 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033633 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033643 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033653 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdjbn\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-kube-api-access-tdjbn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033663 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033676 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033684 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033692 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.033789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.033700 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:42.065961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.065938 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" Oct 31 03:12:42.065961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.065954 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t" event={"ID":"c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0","Type":"ContainerDied","Data":"e76e89d2165aa8f03a1215354b925bc0cb1a5502fbeb514eff1bfc6342b9768f"} Oct 31 03:12:42.067410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.067383 4854 generic.go:356] "Generic (PLEG): container finished" podID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" containerID="94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800" exitCode=0 Oct 31 03:12:42.067531 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.067442 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" Oct 31 03:12:42.067531 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.067492 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" event={"ID":"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa","Type":"ContainerDied","Data":"94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800"} Oct 31 03:12:42.067634 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.067535 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s" event={"ID":"5b785ab8-b765-4ec1-9dc1-ec75dbca43aa","Type":"ContainerDied","Data":"fb4ecee1fa6129e2c14c9b46c5c0f808c4392e66b514cf18768a67e2f986e5c0"} Oct 31 03:12:42.067634 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.067566 4854 scope.go:122] "RemoveContainer" containerID="94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800" Oct 31 03:12:42.079767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.079750 4854 scope.go:122] "RemoveContainer" containerID="94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800" Oct 31 03:12:42.080018 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:12:42.079998 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800\": container with ID starting with 94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800 not found: ID does not exist" containerID="94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800" Oct 31 03:12:42.080099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.080030 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800"} err="failed to get container status \"94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800\": rpc error: code = NotFound desc = could not find container \"94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800\": container with ID starting with 94ea16bfdeef891b373ef623a4fd3c6a8ec5e59ac30f4259f60e1bd3c0c81800 not found: ID does not exist" Oct 31 03:12:42.091323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.091298 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t"] Oct 31 03:12:42.097831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.097802 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1adda1ee-6686694f5d-qfs5t"] Oct 31 03:12:42.114176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114156 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7"] Oct 31 03:12:42.114895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114876 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a10a524c-8cab-4b5e-a354-c02be3852a3a" containerName="payload-processing" Oct 31 03:12:42.114895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114895 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10a524c-8cab-4b5e-a354-c02be3852a3a" containerName="payload-processing" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114927 4854 cpu_manager.go:414] "containerMap: removing container" podUID="eacad21e-7ef5-4042-941b-f2aa7371fb51" containerName="payload-pre-processing" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114936 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacad21e-7ef5-4042-941b-f2aa7371fb51" containerName="payload-pre-processing" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114953 4854 cpu_manager.go:414] "containerMap: removing container" podUID="44b4a004-1a0e-44f3-9861-60e060e23dbf" containerName="payload-processing" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114961 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b4a004-1a0e-44f3-9861-60e060e23dbf" containerName="payload-processing" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.114995 4854 cpu_manager.go:414] "containerMap: removing container" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115001 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115029 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5080e8b7-72c5-47c9-a5e2-3311656d64c3" containerName="payload-pre-processing" Oct 31 03:12:42.115058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115037 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5080e8b7-72c5-47c9-a5e2-3311656d64c3" containerName="payload-pre-processing" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115069 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" containerName="istio-proxy" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115076 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" containerName="istio-proxy" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115088 4854 cpu_manager.go:414] "containerMap: removing container" podUID="625fbabc-a51e-4826-96b0-d34e2f06037f" containerName="maas-api" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115094 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625fbabc-a51e-4826-96b0-d34e2f06037f" containerName="maas-api" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115108 4854 cpu_manager.go:414] "containerMap: removing container" podUID="c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" containerName="maas-api" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115115 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" containerName="maas-api" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115315 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10a524c-8cab-4b5e-a354-c02be3852a3a" containerName="payload-processing" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115334 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacad21e-7ef5-4042-941b-f2aa7371fb51" containerName="payload-pre-processing" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115349 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b4a004-1a0e-44f3-9861-60e060e23dbf" containerName="payload-processing" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115361 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115373 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5080e8b7-72c5-47c9-a5e2-3311656d64c3" containerName="payload-pre-processing" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115387 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" containerName="istio-proxy" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115396 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="625fbabc-a51e-4826-96b0-d34e2f06037f" containerName="maas-api" Oct 31 03:12:42.115501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.115404 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" containerName="maas-api" Oct 31 03:12:42.120498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.120481 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.128185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.128165 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s"] Oct 31 03:12:42.130161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.130142 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bb8bca19\"" Oct 31 03:12:42.132368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.132342 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bb8bca19-dockercfg-gkrhj\"" Oct 31 03:12:42.134785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.134759 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt"] Oct 31 03:12:42.135691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.135662 4854 cpu_manager.go:414] "containerMap: removing container" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" Oct 31 03:12:42.135691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.135683 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" Oct 31 03:12:42.135911 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.135878 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="115985c1-3644-4971-b33f-789bcb827ad3" containerName="istio-proxy" Oct 31 03:12:42.139449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.139430 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7"] Oct 31 03:12:42.139550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.139510 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.144852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.144834 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2ac00dc5-openshift-default-5b8c8f9485-chf8s"] Oct 31 03:12:42.150800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.150780 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt"] Oct 31 03:12:42.281076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.280989 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtknp\" (UniqueName: \"kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp\") pod \"payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.281332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.281309 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume\") pod \"payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.281568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.281549 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.281737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.281722 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vnx\" (UniqueName: \"kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx\") pod \"payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.385514 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.385457 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtknp\" (UniqueName: \"kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp\") pod \"payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.386168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.386141 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume\") pod \"payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.387422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.387281 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume\") pod \"payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.387587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.387404 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.387837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.387786 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vnx\" (UniqueName: \"kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx\") pod \"payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.389716 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.389656 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.411174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.411066 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vnx\" (UniqueName: \"kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx\") pod \"payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.441765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.441522 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:42.512177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.484995 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtknp\" (UniqueName: \"kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp\") pod \"payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.566317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.560122 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115985c1-3644-4971-b33f-789bcb827ad3" path="/var/lib/kubelet/pods/115985c1-3644-4971-b33f-789bcb827ad3/volumes" Oct 31 03:12:42.566317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.562113 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b4a004-1a0e-44f3-9861-60e060e23dbf" path="/var/lib/kubelet/pods/44b4a004-1a0e-44f3-9861-60e060e23dbf/volumes" Oct 31 03:12:42.566317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.562818 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b785ab8-b765-4ec1-9dc1-ec75dbca43aa" path="/var/lib/kubelet/pods/5b785ab8-b765-4ec1-9dc1-ec75dbca43aa/volumes" Oct 31 03:12:42.566317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.563439 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0" path="/var/lib/kubelet/pods/c1ee1396-e3f3-4f4f-a06f-8e8a5cc5c1a0/volumes" Oct 31 03:12:42.569843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.569806 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eacad21e-7ef5-4042-941b-f2aa7371fb51" path="/var/lib/kubelet/pods/eacad21e-7ef5-4042-941b-f2aa7371fb51/volumes" Oct 31 03:12:42.782267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.781981 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:42.896815 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:12:42.896360 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450f689e_2552_4635_aa24_920590727e8f.slice/crio-b78d088aa074f7e992c86e6ce03257e36c9aad31db64ec7a96cc8d0bfd18e472 WatchSource:0}: Error finding container b78d088aa074f7e992c86e6ce03257e36c9aad31db64ec7a96cc8d0bfd18e472: Status 404 returned error can't find the container with id b78d088aa074f7e992c86e6ce03257e36c9aad31db64ec7a96cc8d0bfd18e472 Oct 31 03:12:42.899076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:42.898744 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7"] Oct 31 03:12:43.079731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:43.079692 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" event={"ID":"450f689e-2552-4635-aa24-920590727e8f","Type":"ContainerStarted","Data":"d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314"} Oct 31 03:12:43.079872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:43.079763 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:43.079872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:43.079785 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" event={"ID":"450f689e-2552-4635-aa24-920590727e8f","Type":"ContainerStarted","Data":"b78d088aa074f7e992c86e6ce03257e36c9aad31db64ec7a96cc8d0bfd18e472"} Oct 31 03:12:43.099668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:43.099616 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" podStartSLOduration=1.099603142 podStartE2EDuration="1.099603142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:43.096690912 +0000 UTC m=+1046.793858217" watchObservedRunningTime="2026-10-31 03:12:43.099603142 +0000 UTC m=+1046.796770461" Oct 31 03:12:43.175308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:43.175282 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt"] Oct 31 03:12:43.175873 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:12:43.175851 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode000cf96_0f70_444d_85bd_135ad6c1fcaf.slice/crio-f8c4ed1977416f0366868cfdddfbeb2d413d2d6bf2f2405545dcaa769ee03b60 WatchSource:0}: Error finding container f8c4ed1977416f0366868cfdddfbeb2d413d2d6bf2f2405545dcaa769ee03b60: Status 404 returned error can't find the container with id f8c4ed1977416f0366868cfdddfbeb2d413d2d6bf2f2405545dcaa769ee03b60 Oct 31 03:12:44.099552 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:44.099516 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" event={"ID":"e000cf96-0f70-444d-85bd-135ad6c1fcaf","Type":"ContainerStarted","Data":"8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f"} Oct 31 03:12:44.100074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:44.100042 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" event={"ID":"e000cf96-0f70-444d-85bd-135ad6c1fcaf","Type":"ContainerStarted","Data":"f8c4ed1977416f0366868cfdddfbeb2d413d2d6bf2f2405545dcaa769ee03b60"} Oct 31 03:12:44.100193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:44.100090 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:44.120450 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:44.120405 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" podStartSLOduration=2.120393349 podStartE2EDuration="2.120393349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:44.11619055 +0000 UTC m=+1047.813357855" watchObservedRunningTime="2026-10-31 03:12:44.120393349 +0000 UTC m=+1047.817560652" Oct 31 03:12:45.139182 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.139136 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt"] Oct 31 03:12:45.143752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.143732 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.147050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.147019 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-537aca72-serving-cert\"" Oct 31 03:12:45.166731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.166698 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt"] Oct 31 03:12:45.291603 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.291571 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgrw\" (UniqueName: \"kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.291807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.291785 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.291866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.291847 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.291928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.291915 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.409323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.399933 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.409323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.400075 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgrw\" (UniqueName: \"kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.409323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.400380 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.409323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.400514 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.409323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.401490 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.409323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.408591 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.447273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.446906 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgrw\" (UniqueName: \"kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.452507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.452104 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls\") pod \"maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.460674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.460569 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:45.893694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:45.879910 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt"] Oct 31 03:12:45.900143 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:12:45.900109 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d323ac_f915_4b11_ae4b_62b18601bfc8.slice/crio-d8f5440464a0ca019d1ae3e6ab8d33338117b3578c4ceb1d7eebf5332988127d WatchSource:0}: Error finding container d8f5440464a0ca019d1ae3e6ab8d33338117b3578c4ceb1d7eebf5332988127d: Status 404 returned error can't find the container with id d8f5440464a0ca019d1ae3e6ab8d33338117b3578c4ceb1d7eebf5332988127d Oct 31 03:12:46.116232 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:46.116201 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" event={"ID":"33d323ac-f915-4b11-ae4b-62b18601bfc8","Type":"ContainerStarted","Data":"d8f5440464a0ca019d1ae3e6ab8d33338117b3578c4ceb1d7eebf5332988127d"} Oct 31 03:12:47.135559 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:47.135518 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" event={"ID":"33d323ac-f915-4b11-ae4b-62b18601bfc8","Type":"ContainerStarted","Data":"3020816f91b613261defcb93217e21653baee2c309c53d951445945ec951937e"} Oct 31 03:12:47.137359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:47.137334 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:47.162291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:47.162206 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" podStartSLOduration=1.638744715 podStartE2EDuration="2.162188937s" totalImagesPullingTime="523.444222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:47.158597072 +0000 UTC m=+1050.855764377" watchObservedRunningTime="2026-10-31 03:12:47.162188937 +0000 UTC m=+1050.859356242" Oct 31 03:12:48.533723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.533679 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq"] Oct 31 03:12:48.539610 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.539586 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.550192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.550166 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq"] Oct 31 03:12:48.555970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.555943 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bb8bca19-serving-cert\"" Oct 31 03:12:48.579178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.579155 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.579367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.579340 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.579601 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.579581 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.579753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.579731 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf82m\" (UniqueName: \"kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.681206 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.681178 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.681373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.681291 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.681469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.681454 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.681511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.681484 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf82m\" (UniqueName: \"kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.681916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.681895 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.683928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.683902 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.683928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.683922 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.713973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.713945 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf82m\" (UniqueName: \"kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m\") pod \"maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:48.875331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:48.866539 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:49.625562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:49.625524 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq"] Oct 31 03:12:50.258285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:50.253049 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" event={"ID":"491a4091-42d3-43d8-919c-371c18812c88","Type":"ContainerStarted","Data":"1b0bc28e2c4cae2d8f7b5845aea27131daed51efdab192af26e2c1940b1dc4a8"} Oct 31 03:12:51.546279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.540565 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" event={"ID":"491a4091-42d3-43d8-919c-371c18812c88","Type":"ContainerStarted","Data":"bce266f9d526beedb3e4f9d7d3d196602c7af4c93f233086d9f6fbf12d8621ea"} Oct 31 03:12:51.546279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.541886 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:12:51.590435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.590349 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" podStartSLOduration=2.767780212 podStartE2EDuration="3.590323718s" totalImagesPullingTime="822.543506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:51.588677497 +0000 UTC m=+1055.285844825" watchObservedRunningTime="2026-10-31 03:12:51.590323718 +0000 UTC m=+1055.287491057" Oct 31 03:12:51.699276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.696188 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.731288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.728417 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.731288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.729501 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.761449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.761390 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.990447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.990413 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:12:51.990753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:51.990707 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.502665 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.502044 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.535932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.535366 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.585295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.581462 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.585980 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.585888 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.696450 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.696408 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:12:52.697233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.697208 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:12:52.739391 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.739345 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.740190 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.740165 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:12:52.757409 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.757323 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:12:52.758365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.758342 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:12:52.762340 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.762311 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:12:52.764236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.764208 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:12:52.794370 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.794332 4854 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:12:52.802384 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.802343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.830553 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:52.830508 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.105088 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.104987 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.543579 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.543534 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.544407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.544382 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:12:53.607238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.607190 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:12:53.608076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.608043 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.612337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.608695 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:12:53.612729 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.612538 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:12:53.760274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.713301 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.760274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.744633 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.760274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.758071 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:12:53.792276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.762852 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.877338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.872719 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.877338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.874129 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:12:53.941482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.941389 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:12:53.989092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:53.989049 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.002359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.002330 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:12:54.056920 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.056894 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:12:54.132329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.125514 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.132329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.126302 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:12:54.178416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.178380 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:12:54.183179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.183158 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:12:54.288494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.288465 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:12:54.658273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.613558 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.750274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.716033 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.919804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.919705 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:12:54.938505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:54.935344 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:55.217480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.217378 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:55.331418 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.320560 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:55.426576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.426541 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:12:55.479321 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.468153 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 03:12:55.507932 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.507885 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:12:55.653847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.653810 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:12:55.659170 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.659140 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:12:55.695194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.695162 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:12:55.762044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.761079 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:12:55.762044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.761159 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:12:55.885411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:55.884592 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:12:58.855837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:12:58.855810 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:13:09.252037 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:13:09.251985 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:13:12.815942 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:12.815887 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt"] Oct 31 03:13:12.817492 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:12.817195 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" podUID="33d323ac-f915-4b11-ae4b-62b18601bfc8" containerName="maas-api" containerID="cri-o://3020816f91b613261defcb93217e21653baee2c309c53d951445945ec951937e" gracePeriod=30 Oct 31 03:13:13.413336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.407617 4854 generic.go:356] "Generic (PLEG): container finished" podID="33d323ac-f915-4b11-ae4b-62b18601bfc8" containerID="3020816f91b613261defcb93217e21653baee2c309c53d951445945ec951937e" exitCode=0 Oct 31 03:13:13.413336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.407673 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" event={"ID":"33d323ac-f915-4b11-ae4b-62b18601bfc8","Type":"ContainerDied","Data":"3020816f91b613261defcb93217e21653baee2c309c53d951445945ec951937e"} Oct 31 03:13:13.413336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.407717 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" event={"ID":"33d323ac-f915-4b11-ae4b-62b18601bfc8","Type":"ContainerDied","Data":"d8f5440464a0ca019d1ae3e6ab8d33338117b3578c4ceb1d7eebf5332988127d"} Oct 31 03:13:13.413336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.407737 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f5440464a0ca019d1ae3e6ab8d33338117b3578c4ceb1d7eebf5332988127d" Oct 31 03:13:13.419862 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.419830 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:13:13.528213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.528172 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw\" (UniqueName: \"kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw\") pod \"33d323ac-f915-4b11-ae4b-62b18601bfc8\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " Oct 31 03:13:13.531098 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.531068 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls\") pod \"33d323ac-f915-4b11-ae4b-62b18601bfc8\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " Oct 31 03:13:13.531475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.531454 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs\") pod \"33d323ac-f915-4b11-ae4b-62b18601bfc8\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " Oct 31 03:13:13.531952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.531931 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca\") pod \"33d323ac-f915-4b11-ae4b-62b18601bfc8\" (UID: \"33d323ac-f915-4b11-ae4b-62b18601bfc8\") " Oct 31 03:13:13.538764 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.538733 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca" pod "33d323ac-f915-4b11-ae4b-62b18601bfc8" (UID: "33d323ac-f915-4b11-ae4b-62b18601bfc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:13.633420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.632926 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs" pod "33d323ac-f915-4b11-ae4b-62b18601bfc8" (UID: "33d323ac-f915-4b11-ae4b-62b18601bfc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:13.633420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.633076 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw" pod "33d323ac-f915-4b11-ae4b-62b18601bfc8" (UID: "33d323ac-f915-4b11-ae4b-62b18601bfc8"). InnerVolumeSpecName "kube-api-access-crgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:13.633420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.633136 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls" pod "33d323ac-f915-4b11-ae4b-62b18601bfc8" (UID: "33d323ac-f915-4b11-ae4b-62b18601bfc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:13.635265 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.635215 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:13.635388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.635270 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d323ac-f915-4b11-ae4b-62b18601bfc8-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:13.635388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.635292 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgrw\" (UniqueName: \"kubernetes.io/projected/33d323ac-f915-4b11-ae4b-62b18601bfc8-kube-api-access-crgrw\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:13.635388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:13.635307 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33d323ac-f915-4b11-ae4b-62b18601bfc8-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:14.416923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.416887 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt" Oct 31 03:13:14.440193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.440170 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt"] Oct 31 03:13:14.444559 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.444536 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-537aca72-7f8ccd4979-l4hvt"] Oct 31 03:13:14.458031 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.458009 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d323ac-f915-4b11-ae4b-62b18601bfc8" path="/var/lib/kubelet/pods/33d323ac-f915-4b11-ae4b-62b18601bfc8/volumes" Oct 31 03:13:14.738324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.738290 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b"] Oct 31 03:13:14.739631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.739598 4854 cpu_manager.go:414] "containerMap: removing container" podUID="33d323ac-f915-4b11-ae4b-62b18601bfc8" containerName="maas-api" Oct 31 03:13:14.739631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.739629 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d323ac-f915-4b11-ae4b-62b18601bfc8" containerName="maas-api" Oct 31 03:13:14.740624 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.739847 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d323ac-f915-4b11-ae4b-62b18601bfc8" containerName="maas-api" Oct 31 03:13:14.744582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.744546 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.748898 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.748730 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7692c4-openshift-default-dockercfg-mwn2s\"" Oct 31 03:13:14.756522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.756494 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b"] Oct 31 03:13:14.769856 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.769825 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.769984 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.769884 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.769984 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.769918 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.769984 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.769974 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.770147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.770020 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.770147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.770123 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.770262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.770151 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.770262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.770211 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxsng\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.770403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.770387 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871349 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871458 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871483 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871503 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871546 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871740 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871690 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871802 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871844 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871900 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxsng\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.871996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.871963 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.872099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.872078 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.872243 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.872223 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.872391 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.872375 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.872437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.872414 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.874286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.874224 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.874438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.874416 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.882627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.882602 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:14.883207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:14.883185 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxsng\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng\") pod \"e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:15.066323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:15.065366 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:15.579328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:15.579233 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b"] Oct 31 03:13:15.579328 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:13:15.579303 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc26c32_e947_4d59_9238_abd4cf8d261c.slice/crio-94e0f69b8014388fd874ed057143927c65d75b6d39d0e2ea084e0a082d3fba30 WatchSource:0}: Error finding container 94e0f69b8014388fd874ed057143927c65d75b6d39d0e2ea084e0a082d3fba30: Status 404 returned error can't find the container with id 94e0f69b8014388fd874ed057143927c65d75b6d39d0e2ea084e0a082d3fba30 Oct 31 03:13:15.582110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:15.582077 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:15.582262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:15.582144 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:15.582262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:15.582182 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:16.489627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:16.489592 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" event={"ID":"fbc26c32-e947-4d59-9238-abd4cf8d261c","Type":"ContainerStarted","Data":"059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964"} Oct 31 03:13:16.489835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:16.489646 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" event={"ID":"fbc26c32-e947-4d59-9238-abd4cf8d261c","Type":"ContainerStarted","Data":"94e0f69b8014388fd874ed057143927c65d75b6d39d0e2ea084e0a082d3fba30"} Oct 31 03:13:16.514545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:16.514491 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podStartSLOduration=2.514474939 podStartE2EDuration="2.514474939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:16.510210034 +0000 UTC m=+1080.207377339" watchObservedRunningTime="2026-10-31 03:13:16.514474939 +0000 UTC m=+1080.211642244" Oct 31 03:13:17.070875 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:17.070753 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:17.076970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:17.076945 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:17.150215 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:17.150188 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7"] Oct 31 03:13:17.163189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:17.163162 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt"] Oct 31 03:13:17.535272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:17.534211 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:17.555885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:17.555559 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:22.152944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.151159 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" podUID="450f689e-2552-4635-aa24-920590727e8f" containerName="payload-processing" containerID="cri-o://d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314" gracePeriod=25 Oct 31 03:13:22.179776 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.166313 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" podUID="e000cf96-0f70-444d-85bd-135ad6c1fcaf" containerName="payload-pre-processing" containerID="cri-o://8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f" gracePeriod=25 Oct 31 03:13:22.549421 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.549392 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:13:22.554285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.554231 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:13:22.604676 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.604640 4854 generic.go:356] "Generic (PLEG): container finished" podID="e000cf96-0f70-444d-85bd-135ad6c1fcaf" containerID="8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f" exitCode=0 Oct 31 03:13:22.604886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.604859 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" event={"ID":"e000cf96-0f70-444d-85bd-135ad6c1fcaf","Type":"ContainerDied","Data":"8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f"} Oct 31 03:13:22.605027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.604927 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" event={"ID":"e000cf96-0f70-444d-85bd-135ad6c1fcaf","Type":"ContainerDied","Data":"f8c4ed1977416f0366868cfdddfbeb2d413d2d6bf2f2405545dcaa769ee03b60"} Oct 31 03:13:22.605027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.604942 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt" Oct 31 03:13:22.605027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.604961 4854 scope.go:122] "RemoveContainer" containerID="8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f" Oct 31 03:13:22.606884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.606769 4854 generic.go:356] "Generic (PLEG): container finished" podID="450f689e-2552-4635-aa24-920590727e8f" containerID="d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314" exitCode=0 Oct 31 03:13:22.607039 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.606831 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" Oct 31 03:13:22.607195 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.606846 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" event={"ID":"450f689e-2552-4635-aa24-920590727e8f","Type":"ContainerDied","Data":"d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314"} Oct 31 03:13:22.607369 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.607227 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7" event={"ID":"450f689e-2552-4635-aa24-920590727e8f","Type":"ContainerDied","Data":"b78d088aa074f7e992c86e6ce03257e36c9aad31db64ec7a96cc8d0bfd18e472"} Oct 31 03:13:22.618635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.618617 4854 scope.go:122] "RemoveContainer" containerID="8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f" Oct 31 03:13:22.618930 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:13:22.618912 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f\": container with ID starting with 8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f not found: ID does not exist" containerID="8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f" Oct 31 03:13:22.618987 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.618938 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f"} err="failed to get container status \"8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f\": rpc error: code = NotFound desc = could not find container \"8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f\": container with ID starting with 8b1b7f1f2119dd945d74548234d31e049e2ba4f6032cc963d3cdd6f58a09782f not found: ID does not exist" Oct 31 03:13:22.618987 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.618954 4854 scope.go:122] "RemoveContainer" containerID="d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314" Oct 31 03:13:22.643935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.642442 4854 scope.go:122] "RemoveContainer" containerID="d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314" Oct 31 03:13:22.677482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.675821 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume\") pod \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " Oct 31 03:13:22.677482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.676077 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume\") pod \"450f689e-2552-4635-aa24-920590727e8f\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " Oct 31 03:13:22.677482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.676286 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp\" (UniqueName: \"kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp\") pod \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\" (UID: \"e000cf96-0f70-444d-85bd-135ad6c1fcaf\") " Oct 31 03:13:22.677482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.676364 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx\" (UniqueName: \"kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx\") pod \"450f689e-2552-4635-aa24-920590727e8f\" (UID: \"450f689e-2552-4635-aa24-920590727e8f\") " Oct 31 03:13:22.680481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.679806 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume" pod "450f689e-2552-4635-aa24-920590727e8f" (UID: "450f689e-2552-4635-aa24-920590727e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:22.686336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.680807 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume" pod "e000cf96-0f70-444d-85bd-135ad6c1fcaf" (UID: "e000cf96-0f70-444d-85bd-135ad6c1fcaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:22.692088 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:13:22.692061 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314\": container with ID starting with d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314 not found: ID does not exist" containerID="d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314" Oct 31 03:13:22.692291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.692260 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314"} err="failed to get container status \"d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314\": rpc error: code = NotFound desc = could not find container \"d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314\": container with ID starting with d5c111cd703704ae2fda21818316354061e9ac7938a9ffbed6f5e1da84744314 not found: ID does not exist" Oct 31 03:13:22.730033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.729849 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp" pod "e000cf96-0f70-444d-85bd-135ad6c1fcaf" (UID: "e000cf96-0f70-444d-85bd-135ad6c1fcaf"). InnerVolumeSpecName "kube-api-access-mtknp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:22.788947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.780525 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx" pod "450f689e-2552-4635-aa24-920590727e8f" (UID: "450f689e-2552-4635-aa24-920590727e8f"). InnerVolumeSpecName "kube-api-access-n2vnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:22.789185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.781860 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtknp\" (UniqueName: \"kubernetes.io/projected/e000cf96-0f70-444d-85bd-135ad6c1fcaf-kube-api-access-mtknp\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:22.789345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.789325 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e000cf96-0f70-444d-85bd-135ad6c1fcaf-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:22.789459 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.789447 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f689e-2552-4635-aa24-920590727e8f-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:22.891152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.891115 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vnx\" (UniqueName: \"kubernetes.io/projected/450f689e-2552-4635-aa24-920590727e8f-kube-api-access-n2vnx\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:22.934341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.934243 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt"] Oct 31 03:13:22.938640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.938617 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb8bca19-747d96b875-ffvnt"] Oct 31 03:13:22.946985 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.946960 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7"] Oct 31 03:13:22.953736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:22.953707 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb8bca19-675b69b499-rgml7"] Oct 31 03:13:23.399058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:23.399021 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq"] Oct 31 03:13:23.399621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:23.399282 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" podUID="491a4091-42d3-43d8-919c-371c18812c88" containerName="maas-api" containerID="cri-o://bce266f9d526beedb3e4f9d7d3d196602c7af4c93f233086d9f6fbf12d8621ea" gracePeriod=30 Oct 31 03:13:23.838934 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:23.838816 4854 generic.go:356] "Generic (PLEG): container finished" podID="491a4091-42d3-43d8-919c-371c18812c88" containerID="bce266f9d526beedb3e4f9d7d3d196602c7af4c93f233086d9f6fbf12d8621ea" exitCode=0 Oct 31 03:13:23.842999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:23.839239 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" event={"ID":"491a4091-42d3-43d8-919c-371c18812c88","Type":"ContainerDied","Data":"bce266f9d526beedb3e4f9d7d3d196602c7af4c93f233086d9f6fbf12d8621ea"} Oct 31 03:13:24.087237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.087212 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:13:24.203335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.189914 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m\" (UniqueName: \"kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m\") pod \"491a4091-42d3-43d8-919c-371c18812c88\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " Oct 31 03:13:24.203335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.190021 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca\") pod \"491a4091-42d3-43d8-919c-371c18812c88\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " Oct 31 03:13:24.203335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.190141 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs\") pod \"491a4091-42d3-43d8-919c-371c18812c88\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " Oct 31 03:13:24.284116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.190199 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls\") pod \"491a4091-42d3-43d8-919c-371c18812c88\" (UID: \"491a4091-42d3-43d8-919c-371c18812c88\") " Oct 31 03:13:24.284116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.255979 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca" pod "491a4091-42d3-43d8-919c-371c18812c88" (UID: "491a4091-42d3-43d8-919c-371c18812c88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:24.319993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.314592 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m" pod "491a4091-42d3-43d8-919c-371c18812c88" (UID: "491a4091-42d3-43d8-919c-371c18812c88"). InnerVolumeSpecName "kube-api-access-lf82m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:24.380116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.380046 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf82m\" (UniqueName: \"kubernetes.io/projected/491a4091-42d3-43d8-919c-371c18812c88-kube-api-access-lf82m\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:24.380116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.380084 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491a4091-42d3-43d8-919c-371c18812c88-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:24.381003 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.380956 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs" pod "491a4091-42d3-43d8-919c-371c18812c88" (UID: "491a4091-42d3-43d8-919c-371c18812c88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:24.396273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.392464 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls" pod "491a4091-42d3-43d8-919c-371c18812c88" (UID: "491a4091-42d3-43d8-919c-371c18812c88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:24.481632 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.481529 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:24.482183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.482148 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491a4091-42d3-43d8-919c-371c18812c88-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:24.486814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.486782 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450f689e-2552-4635-aa24-920590727e8f" path="/var/lib/kubelet/pods/450f689e-2552-4635-aa24-920590727e8f/volumes" Oct 31 03:13:24.487317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.487294 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e000cf96-0f70-444d-85bd-135ad6c1fcaf" path="/var/lib/kubelet/pods/e000cf96-0f70-444d-85bd-135ad6c1fcaf/volumes" Oct 31 03:13:24.969145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.968619 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" event={"ID":"491a4091-42d3-43d8-919c-371c18812c88","Type":"ContainerDied","Data":"1b0bc28e2c4cae2d8f7b5845aea27131daed51efdab192af26e2c1940b1dc4a8"} Oct 31 03:13:24.969145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.968843 4854 scope.go:122] "RemoveContainer" containerID="bce266f9d526beedb3e4f9d7d3d196602c7af4c93f233086d9f6fbf12d8621ea" Oct 31 03:13:24.969145 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:24.968967 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" Oct 31 03:13:25.018702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.018666 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq"] Oct 31 03:13:25.118345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.118319 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq"] Oct 31 03:13:25.124589 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.124565 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9"] Oct 31 03:13:25.125895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.125868 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e000cf96-0f70-444d-85bd-135ad6c1fcaf" containerName="payload-pre-processing" Oct 31 03:13:25.127586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.127567 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e000cf96-0f70-444d-85bd-135ad6c1fcaf" containerName="payload-pre-processing" Oct 31 03:13:25.127760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.127745 4854 cpu_manager.go:414] "containerMap: removing container" podUID="450f689e-2552-4635-aa24-920590727e8f" containerName="payload-processing" Oct 31 03:13:25.127850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.127837 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450f689e-2552-4635-aa24-920590727e8f" containerName="payload-processing" Oct 31 03:13:25.127945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.127934 4854 cpu_manager.go:414] "containerMap: removing container" podUID="491a4091-42d3-43d8-919c-371c18812c88" containerName="maas-api" Oct 31 03:13:25.128026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.128016 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491a4091-42d3-43d8-919c-371c18812c88" containerName="maas-api" Oct 31 03:13:25.128318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.128302 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e000cf96-0f70-444d-85bd-135ad6c1fcaf" containerName="payload-pre-processing" Oct 31 03:13:25.128433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.128421 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="450f689e-2552-4635-aa24-920590727e8f" containerName="payload-processing" Oct 31 03:13:25.128533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.128522 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="491a4091-42d3-43d8-919c-371c18812c88" containerName="maas-api" Oct 31 03:13:25.135309 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.135290 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.142140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.141397 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7692c4-serving-cert\"" Oct 31 03:13:25.150885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.150851 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9"] Oct 31 03:13:25.208092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.208055 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.208542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.208518 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfkc\" (UniqueName: \"kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.208726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.208707 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.209007 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.208985 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.316275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.314433 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfkc\" (UniqueName: \"kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.316275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.314541 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.316275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.314713 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.316275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.314811 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.316275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.316062 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.359438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.341826 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.359438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.342766 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.359438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.348222 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfkc\" (UniqueName: \"kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc\") pod \"maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:25.463276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:25.462743 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:26.005240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.004888 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9"] Oct 31 03:13:26.064295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.053236 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh"] Oct 31 03:13:26.089975 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.089345 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh"] Oct 31 03:13:26.089975 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.089464 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.187274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.173628 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-fd2e0b-openshift-default-dockercfg-75hqw\"" Oct 31 03:13:26.239286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216005 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.239286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216057 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.239286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216099 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvgz\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.239286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216176 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.239286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216210 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.239286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216334 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.255386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.216410 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.255386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.247683 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.255386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.247772 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.367676 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.367799 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.367874 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.367994 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.368025 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.368073 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvgz\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.368188 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.368217 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.368409 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.377352 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.380877 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.381149 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.381766 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.382023 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.382567 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.395813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.395760 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvgz\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.402562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.402536 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.458084 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.453997 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token\") pod \"e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.571590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.571507 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491a4091-42d3-43d8-919c-371c18812c88" path="/var/lib/kubelet/pods/491a4091-42d3-43d8-919c-371c18812c88/volumes" Oct 31 03:13:26.626315 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.626284 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:26.864338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:26.863345 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb8bca19-564c5787fc-mglxq" podUID="491a4091-42d3-43d8-919c-371c18812c88" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.99:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:27.036277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:27.030318 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" event={"ID":"521608fc-f682-4cf0-96c2-f529ae646f78","Type":"ContainerStarted","Data":"123e47f66bfffbf51376d521753c1a294ad614850eba0300fdb9793bc802bee8"} Oct 31 03:13:28.051282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.049858 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 31 03:13:28.067280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.061334 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.181231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.177662 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:28.181231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.177761 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:28.181231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.177805 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:28.207976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.207940 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.254453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.221684 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh"] Oct 31 03:13:28.293101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.293077 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn"] Oct 31 03:13:28.298896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.298863 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" event={"ID":"521608fc-f682-4cf0-96c2-f529ae646f78","Type":"ContainerStarted","Data":"e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b"} Oct 31 03:13:28.300416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.300392 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:28.525299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.521145 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:13:28.535287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.532566 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" podStartSLOduration=3.617363823 podStartE2EDuration="4.532527614s" totalImagesPullingTime="915.163791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:28.352650316 +0000 UTC m=+1092.049817632" watchObservedRunningTime="2026-10-31 03:13:28.532527614 +0000 UTC m=+1092.229694920" Oct 31 03:13:28.615333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.605596 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab747d9-lcdbn"] Oct 31 03:13:28.636273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.630391 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:13:28.636273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.630475 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.665274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.662327 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.719326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.718893 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:13:28.737273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.732515 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:13:28.737273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.732562 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.747234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.747194 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.787401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.787329 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:28.953152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:28.953082 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:13:29.083275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.079726 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:13:29.083275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.080434 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:13:29.331414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.331368 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:13:29.332462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.332437 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:13:29.543193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.543148 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:13:29.582275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.558324 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:13:29.634443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.634405 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 31 03:13:29.637021 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.636992 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:13:29.637226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.637202 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:13:29.650358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.650329 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:13:29.650566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.650406 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" event={"ID":"91f59594-de06-478d-a297-6a15040a7353","Type":"ContainerStarted","Data":"4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620"} Oct 31 03:13:29.650566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.650468 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:29.650566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.650491 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" event={"ID":"91f59594-de06-478d-a297-6a15040a7353","Type":"ContainerStarted","Data":"787cfa41733a583c8358a34112cd41abbd1add62182cffce290972cdf49d4a54"} Oct 31 03:13:29.752040 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.751922 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:13:29.798902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.783761 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:13:29.798902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.789344 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:13:29.798902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.790135 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:13:29.798902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.792720 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:13:29.798902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:29.793179 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:13:30.052890 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.052805 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:13:30.059559 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.059528 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:13:30.127800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.127737 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podStartSLOduration=5.127717388 podStartE2EDuration="5.127717388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:30.123662567 +0000 UTC m=+1093.820829883" watchObservedRunningTime="2026-10-31 03:13:30.127717388 +0000 UTC m=+1093.824884694" Oct 31 03:13:30.458751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.458676 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf" path="/var/lib/kubelet/pods/c9c3cc8d-0353-4d49-9afc-e2e30bdc54bf/volumes" Oct 31 03:13:30.661290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.659723 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:30.686683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.686586 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:30.697796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:30.697725 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:13:35.691070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:35.691041 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:13:36.027109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:36.027075 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt"] Oct 31 03:13:36.031872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:36.031847 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a327e88e-g66bt"] Oct 31 03:13:36.517014 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:36.516975 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa71a56-ee52-4860-b3d2-aadd712aa40b" path="/var/lib/kubelet/pods/3fa71a56-ee52-4860-b3d2-aadd712aa40b/volumes" Oct 31 03:13:39.253331 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:13:39.253295 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:13:50.326016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.325980 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v"] Oct 31 03:13:50.332053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.332027 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v"] Oct 31 03:13:50.332231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.332144 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.386276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.382675 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3fc2d0\"" Oct 31 03:13:50.452304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.431482 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3fc2d0-dockercfg-dxktv\"" Oct 31 03:13:50.452578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.436055 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.481272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.453203 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtvp\" (UniqueName: \"kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp\") pod \"payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.641658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.641542 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.674899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.674859 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtvp\" (UniqueName: \"kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp\") pod \"payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.697615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.669031 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.758090 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.758055 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtvp\" (UniqueName: \"kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp\") pod \"payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:50.963799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:50.963709 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:51.348652 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:51.324131 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v"] Oct 31 03:13:51.368314 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:13:51.368279 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eddc677_2ef2_412a_8dda_f2e80581ad59.slice/crio-64fbc576725cb0ff9540eca6415291f8c7b2b69d2b140db74010c55a93ae64bf WatchSource:0}: Error finding container 64fbc576725cb0ff9540eca6415291f8c7b2b69d2b140db74010c55a93ae64bf: Status 404 returned error can't find the container with id 64fbc576725cb0ff9540eca6415291f8c7b2b69d2b140db74010c55a93ae64bf Oct 31 03:13:52.125838 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:52.124495 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" event={"ID":"5eddc677-2ef2-412a-8dda-f2e80581ad59","Type":"ContainerStarted","Data":"1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd"} Oct 31 03:13:52.125838 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:52.124574 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" event={"ID":"5eddc677-2ef2-412a-8dda-f2e80581ad59","Type":"ContainerStarted","Data":"64fbc576725cb0ff9540eca6415291f8c7b2b69d2b140db74010c55a93ae64bf"} Oct 31 03:13:52.125838 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:52.125712 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:13:52.146322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:52.145942 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" podStartSLOduration=2.14592139 podStartE2EDuration="2.14592139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:52.143782538 +0000 UTC m=+1115.840949842" watchObservedRunningTime="2026-10-31 03:13:52.14592139 +0000 UTC m=+1115.843088694" Oct 31 03:13:57.125355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.125274 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj"] Oct 31 03:13:57.172813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.172776 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj"] Oct 31 03:13:57.172998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.172909 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.184928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.184891 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3fc2d0-serving-cert\"" Oct 31 03:13:57.269432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.269401 4854 generic.go:356] "Generic (PLEG): container finished" podID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerID="059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964" exitCode=0 Oct 31 03:13:57.269587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.269474 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" event={"ID":"fbc26c32-e947-4d59-9238-abd4cf8d261c","Type":"ContainerDied","Data":"059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964"} Oct 31 03:13:57.269902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.269885 4854 scope.go:122] "RemoveContainer" containerID="059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964" Oct 31 03:13:57.271447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.271421 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:57.271512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.271484 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:57.271551 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.271515 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:13:57.286888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.286859 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.286998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.286933 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.287050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.287030 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.287103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.287088 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.441389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.418613 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.441389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.418785 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.441389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.418862 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.441389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.418981 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.476447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.475099 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.478055 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.477838 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.597865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.577349 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.659553 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.653044 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:57.825275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:57.792296 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:13:58.342440 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:58.340954 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" event={"ID":"fbc26c32-e947-4d59-9238-abd4cf8d261c","Type":"ContainerStarted","Data":"fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab"} Oct 31 03:13:58.535316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:58.529310 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj"] Oct 31 03:13:59.077420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:59.077339 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:13:59.144275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:59.138183 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:13:59.144275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:59.138242 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 03:13:59.447415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:13:59.447374 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" event={"ID":"37b2af3c-0bd3-4866-b0f0-69572d366e40","Type":"ContainerStarted","Data":"bccbc1435aaa699db1a1d3184d3a706a1f06ceb6d487ec2575cd1070d52b6e76"} Oct 31 03:14:00.065919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:00.065889 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:14:00.066110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:00.065937 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 03:14:00.457766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:00.457681 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" event={"ID":"37b2af3c-0bd3-4866-b0f0-69572d366e40","Type":"ContainerStarted","Data":"4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f"} Oct 31 03:14:00.457766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:00.457736 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:14:00.476390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:00.476344 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" podStartSLOduration=2.6836173690000003 podStartE2EDuration="3.476330848s" totalImagesPullingTime="792.713479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:00.474518794 +0000 UTC m=+1124.171686285" watchObservedRunningTime="2026-10-31 03:14:00.476330848 +0000 UTC m=+1124.173498180" Oct 31 03:14:01.066605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:01.066571 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:14:01.066791 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:01.066621 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 03:14:02.068274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:02.067592 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:14:02.068274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:02.067689 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 03:14:02.533940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:02.533910 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:14:03.066119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:03.066088 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:14:03.066310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:03.066151 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 03:14:04.066100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:04.066061 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:14:04.066596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:04.066119 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 03:14:04.236530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:04.236496 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:14:05.090646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.090608 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:14:05.100198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.100163 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:14:05.706202 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.706174 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw"] Oct 31 03:14:05.710781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.710756 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:05.721103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.721079 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-677281-serving-cert\"" Oct 31 03:14:05.733761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.733736 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw"] Oct 31 03:14:05.898274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.879406 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrx8d\" (UniqueName: \"kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:05.920274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.915998 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:05.920274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.916046 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:05.920274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:05.916204 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.023273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.019612 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrx8d\" (UniqueName: \"kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.023273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.020233 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.023273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.020289 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.023273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.020411 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.023273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.021620 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.046376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.046337 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.047147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.047096 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.052072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.052010 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrx8d\" (UniqueName: \"kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d\") pod \"maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.336957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.336832 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:06.508055 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.508020 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:14:06.831679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:06.830656 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw"] Oct 31 03:14:07.610084 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:07.610051 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" event={"ID":"93aa48af-74e0-458d-8090-2cb2b031f371","Type":"ContainerStarted","Data":"3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5"} Oct 31 03:14:07.610462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:07.610115 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" event={"ID":"93aa48af-74e0-458d-8090-2cb2b031f371","Type":"ContainerStarted","Data":"5eaeafcdf8d19089472d735863431f751b16c9766efdc4f7aa19675332c0f7a9"} Oct 31 03:14:07.610462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:07.610147 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:07.632422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:07.632379 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" podStartSLOduration=2.020700209 podStartE2EDuration="2.632365629s" totalImagesPullingTime="611.66542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:07.62987019 +0000 UTC m=+1131.327037484" watchObservedRunningTime="2026-10-31 03:14:07.632365629 +0000 UTC m=+1131.329532934" Oct 31 03:14:09.283540 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:14:09.283498 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:14:13.648278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:13.643574 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:14:20.584160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:20.583898 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:14:20.722128 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:20.722082 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:14:20.801629 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:20.787887 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 31 03:14:20.963484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:20.963387 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:14:21.117275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:21.108231 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.162271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:21.161488 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.831393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:21.831342 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.906814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:21.832105 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.907893 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:21.886186 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:14:22.064304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:21.967666 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.064304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.061284 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:14:22.077481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.077441 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:14:22.174353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.169236 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.174353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.169829 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:14:22.444231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.444136 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 31 03:14:22.627570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.623483 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.775275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.768711 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.842306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.827836 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:14:22.974457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.974412 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.976444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:22.976404 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:14:23.070350 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.070206 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:23.084283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.079236 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:23.084283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.084048 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:23.109228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.109195 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:14:23.109518 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.109502 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:14:23.432714 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.432523 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:14:23.468057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.467781 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:14:23.553912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.553882 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:14:23.563497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:23.557660 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:14:30.688157 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:30.688062 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:14:30.688157 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:30.688132 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:14:31.089685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:31.089645 4854 generic.go:356] "Generic (PLEG): container finished" podID="91f59594-de06-478d-a297-6a15040a7353" containerID="4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620" exitCode=0 Oct 31 03:14:31.089903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:31.089753 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" event={"ID":"91f59594-de06-478d-a297-6a15040a7353","Type":"ContainerDied","Data":"4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620"} Oct 31 03:14:31.090527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:31.090505 4854 scope.go:122] "RemoveContainer" containerID="4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620" Oct 31 03:14:31.092422 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:31.092396 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:14:31.092499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:31.092463 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:14:31.092537 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:31.092495 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:14:32.124313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:32.124274 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" event={"ID":"91f59594-de06-478d-a297-6a15040a7353","Type":"ContainerStarted","Data":"4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40"} Oct 31 03:14:32.628769 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:32.628736 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:14:32.628956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:32.628932 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:14:32.629035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:32.629004 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:14:33.630597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:33.630559 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:14:33.631236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:33.631201 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:14:34.635290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:34.630878 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:14:34.635880 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:34.635844 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:14:35.629415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:35.629380 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:14:35.629631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:35.629438 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:14:36.629126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:36.629090 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:14:36.629543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:36.629138 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 03:14:37.633704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:37.633676 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:14:37.636069 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:37.636040 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:14:37.637034 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:14:37.637007 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:14:39.371703 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:14:39.371663 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:15:00.260290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.230371 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv"] Oct 31 03:15:00.302013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.301975 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.303955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.303926 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8"] Oct 31 03:15:00.311821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.310899 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.311821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.311042 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv"] Oct 31 03:15:00.315476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.315455 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4"] Oct 31 03:15:00.322606 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.322584 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8"] Oct 31 03:15:00.322725 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.322698 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.324464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.324441 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4"] Oct 31 03:15:00.397032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.397006 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:15:00.475725 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.475693 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.475858 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.475836 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxtk\" (UniqueName: \"kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.475923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.475909 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.475995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.475983 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.476048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.476001 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27bvj\" (UniqueName: \"kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.476190 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.476168 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p89k\" (UniqueName: \"kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.578037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.577948 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.578198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.578106 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxtk\" (UniqueName: \"kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.578198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.578170 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.578374 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.578336 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.578374 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.578367 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27bvj\" (UniqueName: \"kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.578545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.578524 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p89k\" (UniqueName: \"kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.578826 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.578802 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.579117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.579095 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.579117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.579099 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.589907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.589875 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxtk\" (UniqueName: \"kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.590017 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.589996 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p89k\" (UniqueName: \"kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.590059 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.589999 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27bvj\" (UniqueName: \"kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.628662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.628633 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:00.645048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.645026 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:00.662128 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.662102 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:00.920132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.920106 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv"] Oct 31 03:15:00.921119 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:00.921086 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2548b0ff_400c_4452_bef1_816734996189.slice/crio-aeca3454f4c8dd52c60803abfaab585d0c0dea41c37bcd9ca8c8d2ee87dc6eb0 WatchSource:0}: Error finding container aeca3454f4c8dd52c60803abfaab585d0c0dea41c37bcd9ca8c8d2ee87dc6eb0: Status 404 returned error can't find the container with id aeca3454f4c8dd52c60803abfaab585d0c0dea41c37bcd9ca8c8d2ee87dc6eb0 Oct 31 03:15:00.952181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.952158 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8"] Oct 31 03:15:00.952578 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:00.952554 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ab33d7_3973_4684_b69a_2a51e84c366c.slice/crio-1cc3c1aaae71c40f3315b8d5907c7dc65b6d7af54facd455d639cfe201511508 WatchSource:0}: Error finding container 1cc3c1aaae71c40f3315b8d5907c7dc65b6d7af54facd455d639cfe201511508: Status 404 returned error can't find the container with id 1cc3c1aaae71c40f3315b8d5907c7dc65b6d7af54facd455d639cfe201511508 Oct 31 03:15:00.988785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:00.988743 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4"] Oct 31 03:15:00.989507 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:00.989478 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575d7c09_6daa_4d7f_8c24_af8429506175.slice/crio-eeae34dc6315950d19af88295a6038e8c41c3393c7455489c58b0aadcd170f73 WatchSource:0}: Error finding container eeae34dc6315950d19af88295a6038e8c41c3393c7455489c58b0aadcd170f73: Status 404 returned error can't find the container with id eeae34dc6315950d19af88295a6038e8c41c3393c7455489c58b0aadcd170f73 Oct 31 03:15:01.615357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.615322 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerStarted","Data":"aff8bcd25fbe004a8085a7a9f598b07314c6b45a8258616bc8e2bff2ed898481"} Oct 31 03:15:01.615698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.615389 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerStarted","Data":"1cc3c1aaae71c40f3315b8d5907c7dc65b6d7af54facd455d639cfe201511508"} Oct 31 03:15:01.617060 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.617032 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerStarted","Data":"24a8462f28c91bd8b201f5a956d36fc6646db31bf9c3f406d3ab494c43f4d982"} Oct 31 03:15:01.617191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.617082 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerStarted","Data":"aeca3454f4c8dd52c60803abfaab585d0c0dea41c37bcd9ca8c8d2ee87dc6eb0"} Oct 31 03:15:01.618674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.618654 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerStarted","Data":"5099cc92d7e077a7168c7762128e49eeca8245f07e4137b63211827830b3912a"} Oct 31 03:15:01.618750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.618689 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerStarted","Data":"eeae34dc6315950d19af88295a6038e8c41c3393c7455489c58b0aadcd170f73"} Oct 31 03:15:01.652089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.652037 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" podStartSLOduration=1.6520234710000001 podStartE2EDuration="1.652023471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:01.650222962 +0000 UTC m=+1185.347390267" watchObservedRunningTime="2026-10-31 03:15:01.652023471 +0000 UTC m=+1185.349190792" Oct 31 03:15:01.683946 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.683897 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" podStartSLOduration=1.683881581 podStartE2EDuration="1.683881581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:01.680056553 +0000 UTC m=+1185.377223858" watchObservedRunningTime="2026-10-31 03:15:01.683881581 +0000 UTC m=+1185.381048885" Oct 31 03:15:01.730809 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:01.730769 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" podStartSLOduration=1.730757161 podStartE2EDuration="1.730757161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:01.726370104 +0000 UTC m=+1185.423537408" watchObservedRunningTime="2026-10-31 03:15:01.730757161 +0000 UTC m=+1185.427924465" Oct 31 03:15:02.628216 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.628180 4854 generic.go:356] "Generic (PLEG): container finished" podID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerID="aff8bcd25fbe004a8085a7a9f598b07314c6b45a8258616bc8e2bff2ed898481" exitCode=60 Oct 31 03:15:02.628660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.628280 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerDied","Data":"aff8bcd25fbe004a8085a7a9f598b07314c6b45a8258616bc8e2bff2ed898481"} Oct 31 03:15:02.628660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.628579 4854 scope.go:122] "RemoveContainer" containerID="aff8bcd25fbe004a8085a7a9f598b07314c6b45a8258616bc8e2bff2ed898481" Oct 31 03:15:02.630152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.630127 4854 generic.go:356] "Generic (PLEG): container finished" podID="2548b0ff-400c-4452-bef1-816734996189" containerID="24a8462f28c91bd8b201f5a956d36fc6646db31bf9c3f406d3ab494c43f4d982" exitCode=60 Oct 31 03:15:02.630337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.630206 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerDied","Data":"24a8462f28c91bd8b201f5a956d36fc6646db31bf9c3f406d3ab494c43f4d982"} Oct 31 03:15:02.630503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.630400 4854 scope.go:122] "RemoveContainer" containerID="24a8462f28c91bd8b201f5a956d36fc6646db31bf9c3f406d3ab494c43f4d982" Oct 31 03:15:02.632399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.632289 4854 generic.go:356] "Generic (PLEG): container finished" podID="575d7c09-6daa-4d7f-8c24-af8429506175" containerID="5099cc92d7e077a7168c7762128e49eeca8245f07e4137b63211827830b3912a" exitCode=60 Oct 31 03:15:02.632399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.632378 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerDied","Data":"5099cc92d7e077a7168c7762128e49eeca8245f07e4137b63211827830b3912a"} Oct 31 03:15:02.632602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:02.632587 4854 scope.go:122] "RemoveContainer" containerID="5099cc92d7e077a7168c7762128e49eeca8245f07e4137b63211827830b3912a" Oct 31 03:15:03.641401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:03.641319 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerStarted","Data":"7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3"} Oct 31 03:15:03.643294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:03.643227 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerStarted","Data":"82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94"} Oct 31 03:15:03.645434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:03.645413 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerStarted","Data":"444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6"} Oct 31 03:15:04.654971 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.654940 4854 generic.go:356] "Generic (PLEG): container finished" podID="2548b0ff-400c-4452-bef1-816734996189" containerID="82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94" exitCode=60 Oct 31 03:15:04.655465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.655023 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerDied","Data":"82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94"} Oct 31 03:15:04.655465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.655094 4854 scope.go:122] "RemoveContainer" containerID="24a8462f28c91bd8b201f5a956d36fc6646db31bf9c3f406d3ab494c43f4d982" Oct 31 03:15:04.655701 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.655680 4854 scope.go:122] "RemoveContainer" containerID="82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94" Oct 31 03:15:04.656082 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:04.656039 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv_odh-ai-gateway-infra(2548b0ff-400c-4452-bef1-816734996189)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" podUID="2548b0ff-400c-4452-bef1-816734996189" Oct 31 03:15:04.657487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.657466 4854 generic.go:356] "Generic (PLEG): container finished" podID="575d7c09-6daa-4d7f-8c24-af8429506175" containerID="444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6" exitCode=60 Oct 31 03:15:04.657596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.657504 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerDied","Data":"444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6"} Oct 31 03:15:04.657835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.657817 4854 scope.go:122] "RemoveContainer" containerID="444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6" Oct 31 03:15:04.658141 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:04.658115 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4_odh-ai-gateway-infra(575d7c09-6daa-4d7f-8c24-af8429506175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" Oct 31 03:15:04.659680 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.659660 4854 generic.go:356] "Generic (PLEG): container finished" podID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerID="7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3" exitCode=60 Oct 31 03:15:04.659817 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.659721 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerDied","Data":"7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3"} Oct 31 03:15:04.659990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.659971 4854 scope.go:122] "RemoveContainer" containerID="7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3" Oct 31 03:15:04.660234 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:04.660216 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8_odh-ai-gateway-infra(d4ab33d7-3973-4684-b69a-2a51e84c366c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" Oct 31 03:15:04.668818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.668800 4854 scope.go:122] "RemoveContainer" containerID="5099cc92d7e077a7168c7762128e49eeca8245f07e4137b63211827830b3912a" Oct 31 03:15:04.685549 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:04.685533 4854 scope.go:122] "RemoveContainer" containerID="aff8bcd25fbe004a8085a7a9f598b07314c6b45a8258616bc8e2bff2ed898481" Oct 31 03:15:05.670279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:05.670220 4854 scope.go:122] "RemoveContainer" containerID="444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6" Oct 31 03:15:05.670695 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:05.670643 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4_odh-ai-gateway-infra(575d7c09-6daa-4d7f-8c24-af8429506175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" Oct 31 03:15:05.671995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:05.671975 4854 scope.go:122] "RemoveContainer" containerID="7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3" Oct 31 03:15:05.672388 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:05.672365 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8_odh-ai-gateway-infra(d4ab33d7-3973-4684-b69a-2a51e84c366c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" Oct 31 03:15:05.673808 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:05.673780 4854 scope.go:122] "RemoveContainer" containerID="82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94" Oct 31 03:15:05.674070 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:05.674052 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv_odh-ai-gateway-infra(2548b0ff-400c-4452-bef1-816734996189)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" podUID="2548b0ff-400c-4452-bef1-816734996189" Oct 31 03:15:09.251998 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:09.251964 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:15:11.026488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:11.026455 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2"] Oct 31 03:15:11.030460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:11.030430 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc90a75-576f2"] Oct 31 03:15:12.458687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:12.458651 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c42098c-af11-4461-a9ff-a84acc41add0" path="/var/lib/kubelet/pods/7c42098c-af11-4461-a9ff-a84acc41add0/volumes" Oct 31 03:15:15.449332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:15.449306 4854 scope.go:122] "RemoveContainer" containerID="444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6" Oct 31 03:15:15.449714 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:15.449401 4854 scope.go:122] "RemoveContainer" containerID="7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3" Oct 31 03:15:16.700871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.700841 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:15:16.701674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.701653 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:15:16.820900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.820811 4854 generic.go:356] "Generic (PLEG): container finished" podID="575d7c09-6daa-4d7f-8c24-af8429506175" containerID="eedc50dfdeeb2a12e5496a457171d8b3390faa198151c8bf084ce0677be6f52a" exitCode=60 Oct 31 03:15:16.821065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.820892 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerDied","Data":"eedc50dfdeeb2a12e5496a457171d8b3390faa198151c8bf084ce0677be6f52a"} Oct 31 03:15:16.821065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.820957 4854 scope.go:122] "RemoveContainer" containerID="444d90b80ee21562ff6cd163b49a9d7cef82c84302adab0e6e871af8fce889f6" Oct 31 03:15:16.821342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.821321 4854 scope.go:122] "RemoveContainer" containerID="eedc50dfdeeb2a12e5496a457171d8b3390faa198151c8bf084ce0677be6f52a" Oct 31 03:15:16.821651 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:16.821622 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4_odh-ai-gateway-infra(575d7c09-6daa-4d7f-8c24-af8429506175)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" Oct 31 03:15:16.823181 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.823157 4854 generic.go:356] "Generic (PLEG): container finished" podID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerID="a4b7f9a5b719cff9880bfec29945d41ee8ad9096ce02f199cd178eb9750c410a" exitCode=60 Oct 31 03:15:16.823287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.823226 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerDied","Data":"a4b7f9a5b719cff9880bfec29945d41ee8ad9096ce02f199cd178eb9750c410a"} Oct 31 03:15:16.823561 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.823546 4854 scope.go:122] "RemoveContainer" containerID="a4b7f9a5b719cff9880bfec29945d41ee8ad9096ce02f199cd178eb9750c410a" Oct 31 03:15:16.823800 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:16.823782 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8_odh-ai-gateway-infra(d4ab33d7-3973-4684-b69a-2a51e84c366c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" Oct 31 03:15:16.834677 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:16.834662 4854 scope.go:122] "RemoveContainer" containerID="7345d926912fa35f3180bf3cfad9aec226929e1d34012ea045da74296be563d3" Oct 31 03:15:17.449310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:17.449282 4854 scope.go:122] "RemoveContainer" containerID="82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94" Oct 31 03:15:17.836458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:17.836426 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerStarted","Data":"556eaba084b3231f0a27bfc33a1ae8e11c9aa31b2c3d2bcbeb58f823f63127a5"} Oct 31 03:15:17.854016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:17.853989 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4"] Oct 31 03:15:17.867393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:17.867365 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8"] Oct 31 03:15:18.086026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.086000 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:18.089772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.089749 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:18.306979 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.306902 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk\" (UniqueName: \"kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk\") pod \"d4ab33d7-3973-4684-b69a-2a51e84c366c\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " Oct 31 03:15:18.307133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.306986 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca\") pod \"d4ab33d7-3973-4684-b69a-2a51e84c366c\" (UID: \"d4ab33d7-3973-4684-b69a-2a51e84c366c\") " Oct 31 03:15:18.307208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.307160 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca\") pod \"575d7c09-6daa-4d7f-8c24-af8429506175\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " Oct 31 03:15:18.307736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.307699 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca" pod "d4ab33d7-3973-4684-b69a-2a51e84c366c" (UID: "d4ab33d7-3973-4684-b69a-2a51e84c366c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:18.307859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.307709 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca" pod "575d7c09-6daa-4d7f-8c24-af8429506175" (UID: "575d7c09-6daa-4d7f-8c24-af8429506175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:18.307859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.307757 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k\" (UniqueName: \"kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k\") pod \"575d7c09-6daa-4d7f-8c24-af8429506175\" (UID: \"575d7c09-6daa-4d7f-8c24-af8429506175\") " Oct 31 03:15:18.308201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.308175 4854 scope.go:122] "RemoveContainer" containerID="25fb39d46701e582934e88ddede8a0c28574dbbca56172de865a06353c05f517" Oct 31 03:15:18.308492 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.308473 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ab33d7-3973-4684-b69a-2a51e84c366c-maas-api-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.308591 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.308498 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575d7c09-6daa-4d7f-8c24-af8429506175-maas-api-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.310870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.310826 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk" pod "d4ab33d7-3973-4684-b69a-2a51e84c366c" (UID: "d4ab33d7-3973-4684-b69a-2a51e84c366c"). InnerVolumeSpecName "kube-api-access-2vxtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:18.311062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.311038 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k" pod "575d7c09-6daa-4d7f-8c24-af8429506175" (UID: "575d7c09-6daa-4d7f-8c24-af8429506175"). InnerVolumeSpecName "kube-api-access-7p89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:18.341710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.341690 4854 scope.go:122] "RemoveContainer" containerID="6bede7a02733466d6fba4b702a3751fd99234bf8cbcc64c976979606de156bcb" Oct 31 03:15:18.410600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.410576 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p89k\" (UniqueName: \"kubernetes.io/projected/575d7c09-6daa-4d7f-8c24-af8429506175-kube-api-access-7p89k\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.410600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.410601 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxtk\" (UniqueName: \"kubernetes.io/projected/d4ab33d7-3973-4684-b69a-2a51e84c366c-kube-api-access-2vxtk\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.846813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.846722 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" Oct 31 03:15:18.846813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.846748 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8" event={"ID":"d4ab33d7-3973-4684-b69a-2a51e84c366c","Type":"ContainerDied","Data":"1cc3c1aaae71c40f3315b8d5907c7dc65b6d7af54facd455d639cfe201511508"} Oct 31 03:15:18.847339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.846824 4854 scope.go:122] "RemoveContainer" containerID="a4b7f9a5b719cff9880bfec29945d41ee8ad9096ce02f199cd178eb9750c410a" Oct 31 03:15:18.849158 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.849127 4854 generic.go:356] "Generic (PLEG): container finished" podID="2548b0ff-400c-4452-bef1-816734996189" containerID="556eaba084b3231f0a27bfc33a1ae8e11c9aa31b2c3d2bcbeb58f823f63127a5" exitCode=60 Oct 31 03:15:18.849345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.849201 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerDied","Data":"556eaba084b3231f0a27bfc33a1ae8e11c9aa31b2c3d2bcbeb58f823f63127a5"} Oct 31 03:15:18.849711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.849688 4854 scope.go:122] "RemoveContainer" containerID="556eaba084b3231f0a27bfc33a1ae8e11c9aa31b2c3d2bcbeb58f823f63127a5" Oct 31 03:15:18.850081 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:18.850054 4854 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv_odh-ai-gateway-infra(2548b0ff-400c-4452-bef1-816734996189)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" podUID="2548b0ff-400c-4452-bef1-816734996189" Oct 31 03:15:18.850825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.850806 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" event={"ID":"575d7c09-6daa-4d7f-8c24-af8429506175","Type":"ContainerDied","Data":"eeae34dc6315950d19af88295a6038e8c41c3393c7455489c58b0aadcd170f73"} Oct 31 03:15:18.850925 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.850859 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4" Oct 31 03:15:18.859688 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.859471 4854 scope.go:122] "RemoveContainer" containerID="82427b3a070756f5fbab5771bcfc5975ca3ec1bda740f2ce6dfac3bdf98c0f94" Oct 31 03:15:18.870119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.870094 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv"] Oct 31 03:15:18.876814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.876790 4854 scope.go:122] "RemoveContainer" containerID="eedc50dfdeeb2a12e5496a457171d8b3390faa198151c8bf084ce0677be6f52a" Oct 31 03:15:18.910525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.910499 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4"] Oct 31 03:15:18.916505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.916477 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fd2e0b-29890275-rkrs4"] Oct 31 03:15:18.936275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:18.926134 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8"] Oct 31 03:15:19.035575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:19.030835 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-677281-29890275-tczf8"] Oct 31 03:15:20.073041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.073016 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:20.146176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.146095 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj\" (UniqueName: \"kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj\") pod \"2548b0ff-400c-4452-bef1-816734996189\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " Oct 31 03:15:20.146546 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.146522 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca\") pod \"2548b0ff-400c-4452-bef1-816734996189\" (UID: \"2548b0ff-400c-4452-bef1-816734996189\") " Oct 31 03:15:20.146968 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.146944 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca" pod "2548b0ff-400c-4452-bef1-816734996189" (UID: "2548b0ff-400c-4452-bef1-816734996189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:20.147179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.147161 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2548b0ff-400c-4452-bef1-816734996189-maas-api-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:20.149705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.149671 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj" pod "2548b0ff-400c-4452-bef1-816734996189" (UID: "2548b0ff-400c-4452-bef1-816734996189"). InnerVolumeSpecName "kube-api-access-27bvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:20.249461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.249425 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27bvj\" (UniqueName: \"kubernetes.io/projected/2548b0ff-400c-4452-bef1-816734996189-kube-api-access-27bvj\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:20.459922 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.459847 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" path="/var/lib/kubelet/pods/575d7c09-6daa-4d7f-8c24-af8429506175/volumes" Oct 31 03:15:20.460314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.460300 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" path="/var/lib/kubelet/pods/d4ab33d7-3973-4684-b69a-2a51e84c366c/volumes" Oct 31 03:15:20.871625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.871599 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" Oct 31 03:15:20.871625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.871606 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv" event={"ID":"2548b0ff-400c-4452-bef1-816734996189","Type":"ContainerDied","Data":"aeca3454f4c8dd52c60803abfaab585d0c0dea41c37bcd9ca8c8d2ee87dc6eb0"} Oct 31 03:15:20.871857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.871661 4854 scope.go:122] "RemoveContainer" containerID="556eaba084b3231f0a27bfc33a1ae8e11c9aa31b2c3d2bcbeb58f823f63127a5" Oct 31 03:15:20.892763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.892737 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv"] Oct 31 03:15:20.900497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:20.900470 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7692c4-29890275-rrtmv"] Oct 31 03:15:22.459062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:22.459029 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2548b0ff-400c-4452-bef1-816734996189" path="/var/lib/kubelet/pods/2548b0ff-400c-4452-bef1-816734996189/volumes" Oct 31 03:15:25.176981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.176946 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h"] Oct 31 03:15:25.178140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178114 4854 cpu_manager.go:414] "containerMap: removing container" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178140 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178166 4854 cpu_manager.go:414] "containerMap: removing container" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178174 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178190 4854 cpu_manager.go:414] "containerMap: removing container" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178200 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178233 4854 cpu_manager.go:414] "containerMap: removing container" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178240 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178282 4854 cpu_manager.go:414] "containerMap: removing container" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178290 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178307 4854 cpu_manager.go:414] "containerMap: removing container" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178314 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178323 4854 cpu_manager.go:414] "containerMap: removing container" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178330 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178349 4854 cpu_manager.go:414] "containerMap: removing container" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178356 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178536 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178551 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178564 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178577 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178588 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178600 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178613 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ab33d7-3973-4684-b69a-2a51e84c366c" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178623 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.178638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.178634 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="575d7c09-6daa-4d7f-8c24-af8429506175" containerName="cleanup" Oct 31 03:15:25.184747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.184724 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.192541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.192514 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-750c955f-openshift-default-dockercfg-xb2c7\"" Oct 31 03:15:25.193439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.193367 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h"] Oct 31 03:15:25.280205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280177 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280324 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280423 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280479 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280511 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280603 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280587 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280617 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280776 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280761 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.280810 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.280787 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxkt\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.381985 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382099 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382119 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382138 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382405 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382456 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382567 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382676 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.382769 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382717 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxkt\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.383207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382861 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.383207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.382905 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.383436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.383411 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.383639 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.383616 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.383894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.383870 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.385211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.385191 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.385530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.385489 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.394315 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.394290 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.394403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.394384 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxkt\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt\") pod \"e2e-mt-750c955f-openshift-default-7db848f469-hgh7h\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:25.529275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:25.529204 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:26.106820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:26.106695 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h"] Oct 31 03:15:26.114641 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:26.114582 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8621502b_46f5_44a8_8bf2_eccd181a7ee8.slice/crio-3c0c24530b91f543f7b49fb99aacc9c4826cc6ee040f41c8325d805fdd693295 WatchSource:0}: Error finding container 3c0c24530b91f543f7b49fb99aacc9c4826cc6ee040f41c8325d805fdd693295: Status 404 returned error can't find the container with id 3c0c24530b91f543f7b49fb99aacc9c4826cc6ee040f41c8325d805fdd693295 Oct 31 03:15:26.155335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:26.154965 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:15:26.155335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:26.155067 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:15:26.160319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:26.155734 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:15:27.043281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:27.038377 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" event={"ID":"8621502b-46f5-44a8-8bf2-eccd181a7ee8","Type":"ContainerStarted","Data":"24fbc9a25c1ac5ce9e21484bcf35b394dc48ee5540228de635184f7b6e9fa146"} Oct 31 03:15:27.043281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:27.038450 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" event={"ID":"8621502b-46f5-44a8-8bf2-eccd181a7ee8","Type":"ContainerStarted","Data":"3c0c24530b91f543f7b49fb99aacc9c4826cc6ee040f41c8325d805fdd693295"} Oct 31 03:15:27.068275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:27.067676 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podStartSLOduration=2.067654395 podStartE2EDuration="2.067654395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:27.067610252 +0000 UTC m=+1210.764777568" watchObservedRunningTime="2026-10-31 03:15:27.067654395 +0000 UTC m=+1210.764821722" Oct 31 03:15:27.530313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:27.530281 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:27.536449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:27.536422 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:28.125629 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:28.124982 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:28.127379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:28.127348 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:15:30.679366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.679319 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx"] Oct 31 03:15:30.680053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.680036 4854 cpu_manager.go:414] "containerMap: removing container" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:30.680053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.680051 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2548b0ff-400c-4452-bef1-816734996189" containerName="cleanup" Oct 31 03:15:30.684325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.684301 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:30.695720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.695692 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx"] Oct 31 03:15:30.766732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.766693 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-2206e7\"" Oct 31 03:15:30.766926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.766862 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-2206e7-dockercfg-kgxtq\"" Oct 31 03:15:30.801274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.798532 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89cmv\" (UniqueName: \"kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv\") pod \"payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:30.801274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.798617 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:30.980325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.977467 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89cmv\" (UniqueName: \"kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv\") pod \"payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:30.980325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.977561 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:30.980325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:30.978858 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:31.096292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.089824 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89cmv\" (UniqueName: \"kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv\") pod \"payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:31.320275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.315499 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:31.661809 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.661775 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx"] Oct 31 03:15:31.663547 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:31.663523 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38943b33_ed6e_492b_beb8_0eb4bf8a8d81.slice/crio-eed0719aac1aab6e474116d286f0f9cfe7753c001f5a45a19129e6f47591761d WatchSource:0}: Error finding container eed0719aac1aab6e474116d286f0f9cfe7753c001f5a45a19129e6f47591761d: Status 404 returned error can't find the container with id eed0719aac1aab6e474116d286f0f9cfe7753c001f5a45a19129e6f47591761d Oct 31 03:15:31.787998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.787971 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx"] Oct 31 03:15:31.792773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.792751 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:31.803242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.803218 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-750c955f-dockercfg-xp7d9\"" Oct 31 03:15:31.806581 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.806560 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx"] Oct 31 03:15:31.868514 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.868481 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-750c955f\"" Oct 31 03:15:31.908788 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.908753 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wlq\" (UniqueName: \"kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq\") pod \"payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:31.908955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:31.908809 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume\") pod \"payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:32.069002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.060799 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wlq\" (UniqueName: \"kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq\") pod \"payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:32.069002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.060873 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume\") pod \"payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:32.069002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.062059 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume\") pod \"payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:32.245274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.241002 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wlq\" (UniqueName: \"kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq\") pod \"payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:32.360951 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.360849 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" event={"ID":"38943b33-ed6e-492b-beb8-0eb4bf8a8d81","Type":"ContainerStarted","Data":"c52fa572adf9c27cf837a46e3ba61faafc67c4bc13734453b3887e874370fb2b"} Oct 31 03:15:32.360951 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.360947 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" event={"ID":"38943b33-ed6e-492b-beb8-0eb4bf8a8d81","Type":"ContainerStarted","Data":"eed0719aac1aab6e474116d286f0f9cfe7753c001f5a45a19129e6f47591761d"} Oct 31 03:15:32.361207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.361111 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:32.384235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.384192 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" podStartSLOduration=2.38417991 podStartE2EDuration="2.38417991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:32.382122394 +0000 UTC m=+1216.079289699" watchObservedRunningTime="2026-10-31 03:15:32.38417991 +0000 UTC m=+1216.081347214" Oct 31 03:15:32.414921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.414896 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:32.918690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:32.918663 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx"] Oct 31 03:15:32.919099 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:32.918940 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04758a9b_21b7_443c_93c9_fb092dcbc771.slice/crio-f97c4ba233c8fcb56267ef4332cbaff93758c179357e3e73a09a9c73f354ce25 WatchSource:0}: Error finding container f97c4ba233c8fcb56267ef4332cbaff93758c179357e3e73a09a9c73f354ce25: Status 404 returned error can't find the container with id f97c4ba233c8fcb56267ef4332cbaff93758c179357e3e73a09a9c73f354ce25 Oct 31 03:15:33.376619 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:33.376583 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" event={"ID":"04758a9b-21b7-443c-93c9-fb092dcbc771","Type":"ContainerStarted","Data":"6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5"} Oct 31 03:15:33.376792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:33.376643 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" event={"ID":"04758a9b-21b7-443c-93c9-fb092dcbc771","Type":"ContainerStarted","Data":"f97c4ba233c8fcb56267ef4332cbaff93758c179357e3e73a09a9c73f354ce25"} Oct 31 03:15:33.376792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:33.376671 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:33.394952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:33.394907 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" podStartSLOduration=2.394893076 podStartE2EDuration="2.394893076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:33.393511419 +0000 UTC m=+1217.090678724" watchObservedRunningTime="2026-10-31 03:15:33.394893076 +0000 UTC m=+1217.092060380" Oct 31 03:15:39.253154 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:39.253113 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:15:40.614931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.614900 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs"] Oct 31 03:15:40.620443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.620418 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.627895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.627871 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-2206e7-serving-cert\"" Oct 31 03:15:40.636783 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.636755 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs"] Oct 31 03:15:40.707792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.707758 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.708027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.708010 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.708070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.708035 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxczt\" (UniqueName: \"kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.708105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.708075 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.815274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.814543 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.815274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.814591 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxczt\" (UniqueName: \"kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.815274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.814645 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.815274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.814796 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.815274 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:40.815103 4854 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-serving-cert: secret "maas-api-e2e-shared-a-w4-2206e7-serving-cert" not found Oct 31 03:15:40.815274 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:40.815174 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls podName:390be04b-2767-42a6-bed1-4466d0ca43f0 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:41.315151596 +0000 UTC m=+1225.012318898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls") pod "maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" (UID: "390be04b-2767-42a6-bed1-4466d0ca43f0") : secret "maas-api-e2e-shared-a-w4-2206e7-serving-cert" not found Oct 31 03:15:40.825275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.816208 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.849756 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.842148 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:40.850757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:40.850718 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxczt\" (UniqueName: \"kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:41.333276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:41.332399 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:41.351281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:41.349233 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:41.615700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:41.614675 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:42.214390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.209105 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl"] Oct 31 03:15:42.226514 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.226481 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl"] Oct 31 03:15:42.226851 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.226831 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.241298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.239224 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-750c955f-serving-cert\"" Oct 31 03:15:42.259305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.255553 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs"] Oct 31 03:15:42.281432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.279835 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.281432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.279985 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhclf\" (UniqueName: \"kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.281432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.280161 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.281432 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.280281 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.319396 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:42.319354 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390be04b_2767_42a6_bed1_4466d0ca43f0.slice/crio-8485ea790a92ebca88d1389d7686bff04af1b27c73f0312ea647d436a229050f WatchSource:0}: Error finding container 8485ea790a92ebca88d1389d7686bff04af1b27c73f0312ea647d436a229050f: Status 404 returned error can't find the container with id 8485ea790a92ebca88d1389d7686bff04af1b27c73f0312ea647d436a229050f Oct 31 03:15:42.383276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.381939 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.383276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.382020 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.383276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.382299 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.383276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.382388 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhclf\" (UniqueName: \"kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.409274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.383735 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.409274 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:42.383861 4854 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-serving-cert: secret "maas-api-e2e-mt-750c955f-serving-cert" not found Oct 31 03:15:42.409274 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:15:42.383932 4854 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls podName:605d3f71-b3b6-442f-b896-c6b6d1c6ed2b nodeName:}" failed. No retries permitted until 2026-10-31 03:15:42.883911841 +0000 UTC m=+1226.581079124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls") pod "maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" (UID: "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b") : secret "maas-api-e2e-mt-750c955f-serving-cert" not found Oct 31 03:15:42.409274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.408206 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.435269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.434969 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhclf\" (UniqueName: \"kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.738576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.738535 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" event={"ID":"390be04b-2767-42a6-bed1-4466d0ca43f0","Type":"ContainerStarted","Data":"8485ea790a92ebca88d1389d7686bff04af1b27c73f0312ea647d436a229050f"} Oct 31 03:15:42.907873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.907824 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:42.964281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:42.964153 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\") pod \"maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:43.247345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.243204 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:43.379710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.379680 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:15:43.634953 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.634924 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl"] Oct 31 03:15:43.635728 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:43.635697 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605d3f71_b3b6_442f_b896_c6b6d1c6ed2b.slice/crio-701996fa0dceb5d16724544d8bfeecc1b3e2d6d0dc7d98f5971cdc3ae17dbf7d WatchSource:0}: Error finding container 701996fa0dceb5d16724544d8bfeecc1b3e2d6d0dc7d98f5971cdc3ae17dbf7d: Status 404 returned error can't find the container with id 701996fa0dceb5d16724544d8bfeecc1b3e2d6d0dc7d98f5971cdc3ae17dbf7d Oct 31 03:15:43.747749 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.747712 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" event={"ID":"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b","Type":"ContainerStarted","Data":"701996fa0dceb5d16724544d8bfeecc1b3e2d6d0dc7d98f5971cdc3ae17dbf7d"} Oct 31 03:15:43.749577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.749552 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" event={"ID":"390be04b-2767-42a6-bed1-4466d0ca43f0","Type":"ContainerStarted","Data":"d982a94b53f7ec81e9109cb9835e3fcd519df72ce5bf93859c960266a294c9dc"} Oct 31 03:15:43.749775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.749753 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:43.770740 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:43.770693 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" podStartSLOduration=3.093238765 podStartE2EDuration="3.770680319s" totalImagesPullingTime="677.441554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:43.767513629 +0000 UTC m=+1227.464680934" watchObservedRunningTime="2026-10-31 03:15:43.770680319 +0000 UTC m=+1227.467847622" Oct 31 03:15:44.391888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:44.391811 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:15:44.862275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:44.862161 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" event={"ID":"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b","Type":"ContainerStarted","Data":"43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb"} Oct 31 03:15:44.862275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:44.862278 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:44.915307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:44.901782 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" podStartSLOduration=3.424474173 podStartE2EDuration="3.901762956s" totalImagesPullingTime="477.288783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:44.899449313 +0000 UTC m=+1228.596616643" watchObservedRunningTime="2026-10-31 03:15:44.901762956 +0000 UTC m=+1228.598930270" Oct 31 03:15:49.897276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:49.893435 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:15:50.044289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.044240 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s"] Oct 31 03:15:50.055324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.055290 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.067475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.067435 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f1bfbe55-openshift-default-dockercfg-djzcs\"" Oct 31 03:15:50.074441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.074055 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s"] Oct 31 03:15:50.113313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113236 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113339 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113445 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bvq\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113499 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113639 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113623 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113674 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113714 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113788 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.113866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.113855 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.214978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.214904 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.214978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.214945 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215004 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bvq\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215048 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215105 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215136 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215174 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215218 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215310 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215403 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215493 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215532 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.215642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.215619 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.216087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.216063 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.217729 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.217706 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.217957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.217932 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.226698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.226678 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bvq\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.226796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.226678 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.389480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.389445 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:50.739828 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.739795 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s"] Oct 31 03:15:50.786328 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:15:50.783372 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc9ef1f_7391_48a9_af37_186eaa30f981.slice/crio-e44ba543095eb6b51ad1c48b82377095bcfaf2013df6a7a45847a232abbbdb75 WatchSource:0}: Error finding container e44ba543095eb6b51ad1c48b82377095bcfaf2013df6a7a45847a232abbbdb75: Status 404 returned error can't find the container with id e44ba543095eb6b51ad1c48b82377095bcfaf2013df6a7a45847a232abbbdb75 Oct 31 03:15:50.810983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.809478 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:15:50.810983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.809560 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:15:50.810983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.809597 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:15:50.939552 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.939524 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" event={"ID":"0bc9ef1f-7391-48a9-af37-186eaa30f981","Type":"ContainerStarted","Data":"52a79d381256f76392e11da477f674238d53729bf830a6aaf4042d4321ddfc03"} Oct 31 03:15:50.939907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.939607 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" event={"ID":"0bc9ef1f-7391-48a9-af37-186eaa30f981","Type":"ContainerStarted","Data":"e44ba543095eb6b51ad1c48b82377095bcfaf2013df6a7a45847a232abbbdb75"} Oct 31 03:15:50.967070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:50.967016 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podStartSLOduration=0.967001313 podStartE2EDuration="967.001313ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:50.96058781 +0000 UTC m=+1234.657755137" watchObservedRunningTime="2026-10-31 03:15:50.967001313 +0000 UTC m=+1234.664168617" Oct 31 03:15:51.391681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:51.391650 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:51.392501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:51.392473 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:51.392708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:51.392682 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:51.919274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:51.918415 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:15:52.390765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:52.390651 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:52.390765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:52.390711 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:53.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:53.393552 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:53.394271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:53.393608 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:54.398354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:54.394047 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:54.398354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:54.394114 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:55.391278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:55.390863 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:55.391278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:55.390935 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:56.392276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:56.390704 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:56.392276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:56.390764 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:57.414441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:57.414389 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 03:15:57.414938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:57.414463 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 03:15:58.447291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:58.447196 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:58.462158 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:58.449616 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:15:58.479402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:15:58.479368 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:16:01.345283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.344975 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7"] Oct 31 03:16:01.607027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.606631 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.612474 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.612440 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7"] Oct 31 03:16:01.628005 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.621370 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-0a531d-openshift-default-dockercfg-ttbwb\"" Oct 31 03:16:01.854483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.815422 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.854876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.854846 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.855177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.855158 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.855411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.855392 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.855526 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.855513 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.855688 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.855674 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.855824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.855810 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.876367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.856069 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvlw\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.876753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.876721 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.982796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.982741 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.983583 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.983560 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.990743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.990722 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.991345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.991325 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:01.992157 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.992140 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.019278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.992806 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvlw\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.027505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.983444 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.027505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.992071 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.027505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.990662 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.027505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:01.991187 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.052611 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.052577 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.052848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.052823 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvlw\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.077705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.071346 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.077705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.071509 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.077705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.071573 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.077705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.074092 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.110558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.110448 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.144518 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.136801 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token\") pod \"e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.274473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.274439 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:02.930001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.929954 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:02.930582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.930050 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:02.930582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.930091 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:02.930681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:02.930615 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7"] Oct 31 03:16:03.270284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:03.264962 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" event={"ID":"1f0c9ebd-eb44-42ed-b098-874f0511fde9","Type":"ContainerStarted","Data":"4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db"} Oct 31 03:16:03.270594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:03.270561 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" event={"ID":"1f0c9ebd-eb44-42ed-b098-874f0511fde9","Type":"ContainerStarted","Data":"cbae21d8f6fcb5efd941445e2201ec94628cbcaee67ca62c42af68163ad97122"} Oct 31 03:16:03.279878 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:03.279848 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:03.346703 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:03.346648 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podStartSLOduration=2.3466325 podStartE2EDuration="2.3466325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:03.298924014 +0000 UTC m=+1246.996091319" watchObservedRunningTime="2026-10-31 03:16:03.3466325 +0000 UTC m=+1247.043799803" Oct 31 03:16:04.281893 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:04.281864 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:05.281903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:05.281874 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:05.282674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:05.282652 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:16:09.256020 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:16:09.255984 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:16:12.156450 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.156411 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt"] Oct 31 03:16:12.163970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.163943 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.164580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.164554 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt"] Oct 31 03:16:12.175299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.175225 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-780abd-openshift-default-dockercfg-d6r8c\"" Oct 31 03:16:12.289276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.289219 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.289565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.289539 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.289710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.289691 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwdw\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.289890 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.289873 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.290176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.290160 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.290378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.290361 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.290704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.290686 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.290818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.290805 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.290939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.290926 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.403676 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.403782 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.403964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.403993 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.404036 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.404106 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.404152 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.404190 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwdw\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.404272 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.404814 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.405051 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.405281 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.407342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.405791 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.408390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.407446 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.436858 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.436749 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.477294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.474598 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.550275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.543987 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.550275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.542241 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwdw\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw\") pod \"e2e-api-b-780abd-openshift-default-9958655f5-hk4kt\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:12.790106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:12.790070 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:13.264525 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:16:13.264495 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bab621_cecb_4604_b67b_128aa77d685b.slice/crio-409b1508d6976eac8403e921929d49f20e1009b1440483accd0275f7255131ca WatchSource:0}: Error finding container 409b1508d6976eac8403e921929d49f20e1009b1440483accd0275f7255131ca: Status 404 returned error can't find the container with id 409b1508d6976eac8403e921929d49f20e1009b1440483accd0275f7255131ca Oct 31 03:16:13.264525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.264512 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt"] Oct 31 03:16:13.267400 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.267369 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:13.267492 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.267447 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:13.267492 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.267476 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:13.822216 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.822179 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" event={"ID":"c4bab621-cecb-4604-b67b-128aa77d685b","Type":"ContainerStarted","Data":"c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb"} Oct 31 03:16:13.822525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.822426 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" event={"ID":"c4bab621-cecb-4604-b67b-128aa77d685b","Type":"ContainerStarted","Data":"409b1508d6976eac8403e921929d49f20e1009b1440483accd0275f7255131ca"} Oct 31 03:16:13.847676 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:13.847628 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podStartSLOduration=1.847613451 podStartE2EDuration="1.847613451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:13.843680031 +0000 UTC m=+1257.540847336" watchObservedRunningTime="2026-10-31 03:16:13.847613451 +0000 UTC m=+1257.544780755" Oct 31 03:16:14.166480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.166391 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:14.166480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.166456 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:14.794101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.794026 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:14.800121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.800096 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:14.834555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.834525 4854 generic.go:356] "Generic (PLEG): container finished" podID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerID="24fbc9a25c1ac5ce9e21484bcf35b394dc48ee5540228de635184f7b6e9fa146" exitCode=0 Oct 31 03:16:14.834696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.834560 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" event={"ID":"8621502b-46f5-44a8-8bf2-eccd181a7ee8","Type":"ContainerDied","Data":"24fbc9a25c1ac5ce9e21484bcf35b394dc48ee5540228de635184f7b6e9fa146"} Oct 31 03:16:14.834957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.834933 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:14.835062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.835002 4854 scope.go:122] "RemoveContainer" containerID="24fbc9a25c1ac5ce9e21484bcf35b394dc48ee5540228de635184f7b6e9fa146" Oct 31 03:16:14.836211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.835948 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:16:14.836919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.836895 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:14.836965 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.836954 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:14.836999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:14.836985 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:15.845966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:15.845932 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" event={"ID":"8621502b-46f5-44a8-8bf2-eccd181a7ee8","Type":"ContainerStarted","Data":"188d359394a320293e0deb007e4d7372627e98a307716f9aa893b5b12dca5a2a"} Oct 31 03:16:16.530229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:16.530198 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:16:16.530409 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:16.530389 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:16.530470 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:16.530451 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:17.530336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:17.530304 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:17.530750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:17.530365 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:18.377748 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:18.377032 4854 scope.go:122] "RemoveContainer" containerID="587b89e072f02fed9d2638bc6e77cc514b21ad1145cdc906d126f5b7fb1128ea" Oct 31 03:16:18.530586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:18.530555 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:18.532293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:18.531739 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:19.559287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:19.557370 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:19.559287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:19.557437 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:20.537114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:20.533356 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:20.537114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:20.533430 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:21.530479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:21.530444 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:21.530874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:21.530492 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:22.530649 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:22.530608 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 03:16:22.531051 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:22.530683 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 03:16:23.564606 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:23.564574 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:16:23.572227 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:23.572182 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:16:23.575707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:23.575678 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:16:25.489341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.489302 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94"] Oct 31 03:16:25.502399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.502359 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.507403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.507380 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94"] Oct 31 03:16:25.511343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.511319 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2206e7-openshift-default-dockercfg-t49h2\"" Oct 31 03:16:25.548930 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.548897 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.548930 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.548932 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549142 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549028 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549166 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487vq\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549214 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549398 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549534 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549556 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.549625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.549611 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651237 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651363 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651380 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651413 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651589 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651596 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651650 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651671 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651705 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651732 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487vq\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651770 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.651938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651777 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.652046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.651940 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.652489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.652469 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.654079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.654061 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.654361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.654342 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.663128 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.663107 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.663263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.663133 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487vq\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:25.830322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:25.829659 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:26.751593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:26.751559 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94"] Oct 31 03:16:26.849633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:26.849578 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:26.849897 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:26.849873 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:26.850072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:26.850046 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:16:27.365135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:27.365030 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" event={"ID":"7ae6cf33-76ca-40b1-9424-59dadb8de118","Type":"ContainerStarted","Data":"0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623"} Oct 31 03:16:27.365135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:27.365105 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" event={"ID":"7ae6cf33-76ca-40b1-9424-59dadb8de118","Type":"ContainerStarted","Data":"d42492e5a9aa1757a594f257fb5045c77ef416b20491811235c075d0a96e3661"} Oct 31 03:16:27.394708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:27.394131 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podStartSLOduration=2.394114106 podStartE2EDuration="2.394114106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:27.388491661 +0000 UTC m=+1271.085658968" watchObservedRunningTime="2026-10-31 03:16:27.394114106 +0000 UTC m=+1271.091281414" Oct 31 03:16:27.836014 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:27.835983 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:29.108276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.077924 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:16:29.108276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.077981 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 03:16:29.475455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.475354 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:16:29.662275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.661685 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:16:29.662632 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.662540 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.666271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.664822 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:16:29.718271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.707621 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.751325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.738162 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.776283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.752213 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.826283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.794688 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.861408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.799737 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.916400 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.845506 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:16:29.916713 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.916676 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 03:16:29.916904 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:29.847906 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:16:30.349278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.347393 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.458083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.458037 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.476763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.476719 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.488816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.488789 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:16:30.664440 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.664337 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:16:30.665232 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.665206 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:16:30.717436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.688399 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.717436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.688921 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:16:30.824853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.824810 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.826589 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.826563 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:16:30.827997 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.826779 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 03:16:30.828547 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.828527 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:16:30.830585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.830566 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:16:30.830733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.830714 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 03:16:30.856436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.856302 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.927338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.915411 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:30.927338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:30.915755 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:16:31.097026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.094722 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.499272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.487442 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:16:31.499272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.490318 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.499272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.490838 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:16:31.549857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.549740 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:16:31.557530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.557498 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.590901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.590872 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:16:31.685318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.679003 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:16:31.689661 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.689619 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.756332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.753107 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:16:31.837390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.837346 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.842511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.837348 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.843032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.837821 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:16:31.846282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.846113 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:16:31.846282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.846157 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 03:16:31.888480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.888450 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:16:31.911800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:31.911081 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:16:32.841323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:32.832115 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:16:32.847632 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:32.847343 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 03:16:33.830874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:33.830835 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:16:33.831156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:33.830895 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 03:16:34.911213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:34.911183 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:34.912044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:34.912020 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:34.912916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:16:34.912895 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:16:39.354725 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:16:39.354686 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:17:08.703295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:08.692275 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:17:08.846212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:08.834384 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:08.846212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:08.844130 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:08.890278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:08.877787 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:08.898387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:08.898340 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:09.399576 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:17:09.399530 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:17:09.667689 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:09.667586 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:09.695404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:09.695338 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:09.700827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:09.700800 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:17:09.925330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:09.925064 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:17:09.925873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:09.925411 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:17:09.953677 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:09.926055 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:17:10.393262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.393194 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.557274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.554337 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.618708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.618002 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:17:10.752332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.716083 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.779275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.767054 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.837434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.837373 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.941761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.940136 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:17:10.943820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.943787 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.952180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.952147 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.955410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.955390 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:17:10.975470 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.975413 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:10.992463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:10.982890 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:11.077519 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.077430 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:11.494399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.494366 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kc577 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:17:11.494658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.494627 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kc577" podUID="3621ef10-eb7d-4e57-9c29-5f3598cae2d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:11.603279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.579888 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:11.603279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.580469 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:17:11.621861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.621821 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:17:11.640275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.622610 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:17:11.859455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.858563 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:11.859455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.858953 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:17:11.880275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.860065 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:17:11.970845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.945510 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:11.970845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.946172 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:17:11.987355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.985383 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:17:12.003293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.987628 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:12.003293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.988670 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:17:12.003293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.994912 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:12.003293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:11.996307 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:17:12.053278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:12.021353 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:12.604329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:12.602228 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:12.606438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:12.606238 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 31 03:17:12.936330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:12.862116 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:12.936330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:12.862355 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:17:12.936330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:12.862460 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:17:13.081287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.079332 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:13.505936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.505883 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:13.562277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.531139 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:13.642752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.642713 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:17:13.690680 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.690633 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:13.717270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.714154 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:17:13.717270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.714212 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:13.782358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.754569 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:13.853313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.782438 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:17:13.853313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:13.845282 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:14.041760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.041655 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:17:14.082015 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.081970 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jlgpj" podUID="9ca899b4-82ad-4c0f-9654-f9912c269a4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Oct 31 03:17:14.086372 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.086336 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:17:14.139754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.102360 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:17:14.326182 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:17:14.326092 4854 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.874s" Oct 31 03:17:14.545441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.545392 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:17:14.584791 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.555423 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:17:14.586844 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.555481 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 31 03:17:14.691283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.688769 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:17:14.738276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.698690 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:14.848320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.826977 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:14.848320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.845412 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:14.876967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:14.876936 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:17:15.121330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.119639 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:17:15.145675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.137480 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:17:15.196271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.190111 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:15.418333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.383464 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:15.604303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.598357 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:15.604303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.598714 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 31 03:17:15.699375 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.689286 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:15.700218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.700193 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68"] Oct 31 03:17:15.748316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.745907 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:17:15.813511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.813471 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68"] Oct 31 03:17:15.813887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.813862 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:15.907771 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.838598 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:15.907771 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.839876 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b93a15-dockercfg-g82ch\"" Oct 31 03:17:15.907771 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.875712 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b93a15\"" Oct 31 03:17:15.949275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.913627 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz"] Oct 31 03:17:15.980738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:15.968576 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.069807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.069765 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.070286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.070239 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgwz\" (UniqueName: \"kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz\") pod \"payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.124382 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.124335 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:16.154383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.154331 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz"] Oct 31 03:17:16.173079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.173038 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.173394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.173375 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.173605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.173590 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqqn\" (UniqueName: \"kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn\") pod \"payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.173759 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.173739 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgwz\" (UniqueName: \"kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz\") pod \"payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.175410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.175384 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.184196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.184178 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:17:16.191450 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.191417 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:16.269328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.266593 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgwz\" (UniqueName: \"kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz\") pod \"payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.311280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.282964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.312024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.311990 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqqn\" (UniqueName: \"kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn\") pod \"payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.349545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.313090 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.388466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.313539 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:16.592970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.589330 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqqn\" (UniqueName: \"kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn\") pod \"payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.691026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.690987 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b93a15-dockercfg-g82ch\"" Oct 31 03:17:16.787275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.785053 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:16.818439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.818411 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:16.853413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.820704 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:17:16.853413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.828930 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:16.982799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.982762 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:16.988376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:16.988347 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:17:17.109578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.086232 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:17:17.109578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.086343 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:17:17.109578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.086706 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:17:17.174880 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.174817 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:17:17.183460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.182896 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:17:17.670416 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.670389 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz"] Oct 31 03:17:18.008166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:17.971352 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68"] Oct 31 03:17:18.027350 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:18.027316 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" event={"ID":"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9","Type":"ContainerStarted","Data":"4afdea19a5b378f3991db085995d5835ac14d1501752bd1ad0bdb306e8f7e753"} Oct 31 03:17:18.029446 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:18.029416 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" event={"ID":"4db2452b-ccf0-41a2-a16a-a1fde0d1da65","Type":"ContainerStarted","Data":"4541684a89b0228a6c2883c53d449cf30f43f4e9554666eafbbb12a978918166"} Oct 31 03:17:19.040233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:19.040199 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" event={"ID":"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9","Type":"ContainerStarted","Data":"ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3"} Oct 31 03:17:19.040719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:19.040598 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:19.042431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:19.042401 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" event={"ID":"4db2452b-ccf0-41a2-a16a-a1fde0d1da65","Type":"ContainerStarted","Data":"cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94"} Oct 31 03:17:19.042561 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:19.042536 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:19.059946 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:19.059792 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" podStartSLOduration=6.059777265 podStartE2EDuration="6.059777265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:19.058981681 +0000 UTC m=+1322.756148981" watchObservedRunningTime="2026-10-31 03:17:19.059777265 +0000 UTC m=+1322.756944570" Oct 31 03:17:19.076968 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:19.076878 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" podStartSLOduration=6.076866501 podStartE2EDuration="6.076866501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:19.076038858 +0000 UTC m=+1322.773206174" watchObservedRunningTime="2026-10-31 03:17:19.076866501 +0000 UTC m=+1322.774033805" Oct 31 03:17:21.327419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:21.327373 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:17:21.327830 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:21.327451 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:24.093801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.093718 4854 generic.go:356] "Generic (PLEG): container finished" podID="c4bab621-cecb-4604-b67b-128aa77d685b" containerID="c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb" exitCode=0 Oct 31 03:17:24.093801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.093788 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" event={"ID":"c4bab621-cecb-4604-b67b-128aa77d685b","Type":"ContainerDied","Data":"c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb"} Oct 31 03:17:24.094381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.094359 4854 scope.go:122] "RemoveContainer" containerID="c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb" Oct 31 03:17:24.096074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.096047 4854 generic.go:356] "Generic (PLEG): container finished" podID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerID="4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db" exitCode=0 Oct 31 03:17:24.096168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.096099 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" event={"ID":"1f0c9ebd-eb44-42ed-b098-874f0511fde9","Type":"ContainerDied","Data":"4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db"} Oct 31 03:17:24.096281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.096211 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:17:24.096382 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.096335 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:17:24.096382 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.096373 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:17:24.096522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.096507 4854 scope.go:122] "RemoveContainer" containerID="4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db" Oct 31 03:17:24.098133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.098105 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:17:24.098217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.098168 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:17:24.098217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:24.098205 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:17:25.187050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.187012 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" event={"ID":"1f0c9ebd-eb44-42ed-b098-874f0511fde9","Type":"ContainerStarted","Data":"981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d"} Oct 31 03:17:25.311476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.307708 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:17:25.311476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.308371 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:25.311476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.308460 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:25.517388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.498215 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" event={"ID":"c4bab621-cecb-4604-b67b-128aa77d685b","Type":"ContainerStarted","Data":"7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e"} Oct 31 03:17:25.800379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.798609 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:17:25.834274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.815301 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:25.834274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:25.815419 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:26.336279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:26.333668 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:26.336279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:26.333757 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:26.804272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:26.795561 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:26.825902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:26.825865 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:27.284278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:27.283348 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:27.284278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:27.283420 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:27.798464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:27.798430 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:27.799135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:27.799101 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:28.275472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:28.275439 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:28.275648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:28.275497 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:28.797641 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:28.797601 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:28.797928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:28.797897 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:29.275132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.275098 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:29.314274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.300946 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:29.414287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.401131 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m"] Oct 31 03:17:29.415129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.415094 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" containerID="cri-o://40e6cad9a2033b9dc41a884579d9affcf0f67f42c133913925fdda001952c8a2" gracePeriod=30 Oct 31 03:17:29.553628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.552706 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj"] Oct 31 03:17:29.553628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.553390 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" containerID="cri-o://03a16040f8fd1834935dbfc20cc0fcc31821c15a1ded986b82a675814f8505c8" gracePeriod=30 Oct 31 03:17:29.622660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.622589 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp"] Oct 31 03:17:29.623604 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.623568 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" containerID="cri-o://b53514ece8e6707029b8e60d57649085b7e606f4cac910fed9ad6ee99af5e20f" gracePeriod=30 Oct 31 03:17:29.796420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.796381 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:29.796742 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.796707 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:29.839493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:29.839392 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:17:30.060303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:30.060175 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:17:30.063802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:30.063776 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:17:30.275739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:30.275705 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:30.276110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:30.275764 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:30.815272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:30.811031 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:30.815272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:30.811097 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:31.292283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:31.282136 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:31.322404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:31.300822 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:31.324178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:31.293080 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:31.488755 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:31.488724 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m"] Oct 31 03:17:31.804805 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:31.804771 4854 patch_prober.go:29] interesting pod/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:17:31.805076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:31.805046 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 03:17:32.275002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:32.274973 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 03:17:32.275183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:32.275022 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 03:17:32.831105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:32.831060 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:17:32.852287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:32.848570 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:17:33.280030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:33.279995 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:17:33.280481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:33.280466 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:17:33.281143 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:33.281121 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:17:34.983366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:34.983338 4854 generic.go:356] "Generic (PLEG): container finished" podID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerID="03a16040f8fd1834935dbfc20cc0fcc31821c15a1ded986b82a675814f8505c8" exitCode=0 Oct 31 03:17:34.983836 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:34.983414 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" event={"ID":"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8","Type":"ContainerDied","Data":"03a16040f8fd1834935dbfc20cc0fcc31821c15a1ded986b82a675814f8505c8"} Oct 31 03:17:34.985987 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:34.985963 4854 generic.go:356] "Generic (PLEG): container finished" podID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerID="b53514ece8e6707029b8e60d57649085b7e606f4cac910fed9ad6ee99af5e20f" exitCode=0 Oct 31 03:17:34.986131 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:34.986110 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" event={"ID":"6897173b-527e-4104-8248-eb37e8f3b3c0","Type":"ContainerDied","Data":"b53514ece8e6707029b8e60d57649085b7e606f4cac910fed9ad6ee99af5e20f"} Oct 31 03:17:34.989039 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:34.989021 4854 generic.go:356] "Generic (PLEG): container finished" podID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerID="40e6cad9a2033b9dc41a884579d9affcf0f67f42c133913925fdda001952c8a2" exitCode=0 Oct 31 03:17:34.989319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:34.989135 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" event={"ID":"17a8a06b-ea59-4eaa-b20f-99bde34963a0","Type":"ContainerDied","Data":"40e6cad9a2033b9dc41a884579d9affcf0f67f42c133913925fdda001952c8a2"} Oct 31 03:17:35.219577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.219503 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:17:35.234104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.233529 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:17:35.339004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.338971 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.339166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339027 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.339166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339057 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.339166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339108 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.339166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339137 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.339394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339306 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.339394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339365 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.339588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339566 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.339643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339628 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.339697 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339659 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.339751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339729 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.339804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339796 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.339865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.339847 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.340094 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340074 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.340154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340112 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.340221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340204 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket\") pod \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\" (UID: \"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8\") " Oct 31 03:17:35.340319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340287 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.340410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340392 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.340552 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340536 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.340637 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340621 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2\") pod \"6897173b-527e-4104-8248-eb37e8f3b3c0\" (UID: \"6897173b-527e-4104-8248-eb37e8f3b3c0\") " Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.340974 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341165 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341465 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341740 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341761 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341777 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341792 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.342027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.341805 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.342694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.342667 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.342919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.342899 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.343425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.343188 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:35.343577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.343552 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.344138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.344114 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:35.344628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.344602 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "kube-api-access-n55gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.348388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.348363 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:35.348498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.348426 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:35.349724 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.349700 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.350324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.350242 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.350421 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.350398 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "kube-api-access-rnbd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.351706 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.351134 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy" pod "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" (UID: "5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.355582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.355558 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy" pod "6897173b-527e-4104-8248-eb37e8f3b3c0" (UID: "6897173b-527e-4104-8248-eb37e8f3b3c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.424211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.424183 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:17:35.442633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.442604 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.442794 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.442641 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.442794 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.442681 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.442794 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.442740 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.442794 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.442771 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.443029 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.442814 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.443092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.443065 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.443152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.443100 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.443230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.443187 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket\") pod \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\" (UID: \"17a8a06b-ea59-4eaa-b20f-99bde34963a0\") " Oct 31 03:17:35.444117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444035 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.444317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444279 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.444400 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444358 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.444499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444438 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.444618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444595 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:35.444958 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444934 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444964 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnbd2\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-kube-api-access-rnbd2\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444979 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.444991 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445004 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445017 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445029 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445042 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445055 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445067 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55gm\" (UniqueName: \"kubernetes.io/projected/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-kube-api-access-n55gm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445081 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445095 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445107 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445120 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445133 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897173b-527e-4104-8248-eb37e8f3b3c0-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445145 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445157 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897173b-527e-4104-8248-eb37e8f3b3c0-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445169 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.445444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445274 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "kube-api-access-ssxhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.445926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.445872 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.447169 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.447142 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:35.447505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.447324 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token" pod "17a8a06b-ea59-4eaa-b20f-99bde34963a0" (UID: "17a8a06b-ea59-4eaa-b20f-99bde34963a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.546328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.546288 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.546328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.546321 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxhc\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-kube-api-access-ssxhc\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.546328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.546336 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.546657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.546349 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a8a06b-ea59-4eaa-b20f-99bde34963a0-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:36.000002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.999969 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" Oct 31 03:17:36.000554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:35.999994 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj" event={"ID":"5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8","Type":"ContainerDied","Data":"850d22080068a6565a9fa323deb151ae9d4d5af63788c195b5c623dd0477544c"} Oct 31 03:17:36.000554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.000062 4854 scope.go:122] "RemoveContainer" containerID="03a16040f8fd1834935dbfc20cc0fcc31821c15a1ded986b82a675814f8505c8" Oct 31 03:17:36.001818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.001804 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" Oct 31 03:17:36.001919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.001845 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp" event={"ID":"6897173b-527e-4104-8248-eb37e8f3b3c0","Type":"ContainerDied","Data":"8902f09dba7116621413e6b0ff748e2f8952026cd8a3ce95a5cb31c5e3f8c3eb"} Oct 31 03:17:36.003613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.003591 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" event={"ID":"17a8a06b-ea59-4eaa-b20f-99bde34963a0","Type":"ContainerDied","Data":"a1d8de4df6b4fec1ee0987d550a0c046540a6f351605b128c73ba6e10eb10b61"} Oct 31 03:17:36.003731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.003641 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m" Oct 31 03:17:36.017073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.015884 4854 scope.go:122] "RemoveContainer" containerID="b53514ece8e6707029b8e60d57649085b7e606f4cac910fed9ad6ee99af5e20f" Oct 31 03:17:36.034323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.034300 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj"] Oct 31 03:17:36.034774 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.034754 4854 scope.go:122] "RemoveContainer" containerID="40e6cad9a2033b9dc41a884579d9affcf0f67f42c133913925fdda001952c8a2" Oct 31 03:17:36.043717 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.043683 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-fh7qj"] Oct 31 03:17:36.060457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.060435 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp"] Oct 31 03:17:36.066981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.066961 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-tnfsp"] Oct 31 03:17:36.077193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.077174 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m"] Oct 31 03:17:36.083396 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.083375 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-k882m"] Oct 31 03:17:36.461561 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.461489 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" path="/var/lib/kubelet/pods/17a8a06b-ea59-4eaa-b20f-99bde34963a0/volumes" Oct 31 03:17:36.461979 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.461965 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" path="/var/lib/kubelet/pods/5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8/volumes" Oct 31 03:17:36.462427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.462414 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" path="/var/lib/kubelet/pods/6897173b-527e-4104-8248-eb37e8f3b3c0/volumes" Oct 31 03:17:36.941205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:36.941174 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r"] Oct 31 03:17:39.257605 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:17:39.257568 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:17:41.942584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:41.942540 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerName="payload-processing" containerID="cri-o://9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726" gracePeriod=25 Oct 31 03:17:42.719170 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.719138 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp"] Oct 31 03:17:42.720387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720362 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" Oct 31 03:17:42.720387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720386 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" Oct 31 03:17:42.720541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720420 4854 cpu_manager.go:414] "containerMap: removing container" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" Oct 31 03:17:42.720541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720429 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" Oct 31 03:17:42.720541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720466 4854 cpu_manager.go:414] "containerMap: removing container" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" Oct 31 03:17:42.720541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720474 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" Oct 31 03:17:42.720671 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720659 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9439fc-31f9-40a0-9ac8-e27d7e6c43e8" containerName="istio-proxy" Oct 31 03:17:42.720702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720675 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="6897173b-527e-4104-8248-eb37e8f3b3c0" containerName="istio-proxy" Oct 31 03:17:42.720702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.720685 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a8a06b-ea59-4eaa-b20f-99bde34963a0" containerName="istio-proxy" Oct 31 03:17:42.738971 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.733522 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.738971 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.737000 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp"] Oct 31 03:17:42.739973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.739949 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f1bfbe55-serving-cert\"" Oct 31 03:17:42.814947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.814902 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.815088 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.815050 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.815088 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.815070 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8xx\" (UniqueName: \"kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.815305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.815286 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.940280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.939583 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.940280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.939743 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.940280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.939814 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8xx\" (UniqueName: \"kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.940280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.940032 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.979326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.958350 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:42.979326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:42.977692 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:43.012462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:43.012384 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:43.016988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:43.016918 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8xx\" (UniqueName: \"kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:43.072711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:43.068630 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:43.128012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:43.111711 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx"] Oct 31 03:17:43.725293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:43.724600 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:17:43.760615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:43.760525 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp"] Oct 31 03:17:44.149584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:44.149503 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" event={"ID":"6a8ffab6-dfb1-4979-8090-d718a41aad1f","Type":"ContainerStarted","Data":"6463e6df2c1abe40003e461ea46161b8998664ef69148d7d71990e29d6504547"} Oct 31 03:17:45.372576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:45.372526 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" event={"ID":"6a8ffab6-dfb1-4979-8090-d718a41aad1f","Type":"ContainerStarted","Data":"de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004"} Oct 31 03:17:45.374616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:45.374593 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:46.624700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:46.623872 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" containerID="cri-o://a3beddf22dfee12a7dacafff7bbaf19b3165b4b35acb7612f7c072a0e4422aab" gracePeriod=45 Oct 31 03:17:47.080610 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.080569 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused" Oct 31 03:17:47.486399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.476004 4854 generic.go:356] "Generic (PLEG): container finished" podID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerID="a3beddf22dfee12a7dacafff7bbaf19b3165b4b35acb7612f7c072a0e4422aab" exitCode=0 Oct 31 03:17:47.486399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.477090 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" event={"ID":"9d6bd592-ed4c-4a14-9455-fbdd18419ad8","Type":"ContainerDied","Data":"a3beddf22dfee12a7dacafff7bbaf19b3165b4b35acb7612f7c072a0e4422aab"} Oct 31 03:17:47.640882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.640842 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:17:47.662733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.662633 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" podStartSLOduration=5.180073472 podStartE2EDuration="5.662617425s" totalImagesPullingTime="482.543953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:45.393457859 +0000 UTC m=+1349.090625164" watchObservedRunningTime="2026-10-31 03:17:47.662617425 +0000 UTC m=+1351.359784730" Oct 31 03:17:47.735427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735400 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir\") pod \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " Oct 31 03:17:47.735580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735456 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home\") pod \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " Oct 31 03:17:47.735580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735498 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l\" (UniqueName: \"kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l\") pod \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " Oct 31 03:17:47.735782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735589 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs\") pod \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " Oct 31 03:17:47.735782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735647 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm\") pod \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " Oct 31 03:17:47.735782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735700 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache\") pod \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\" (UID: \"9d6bd592-ed4c-4a14-9455-fbdd18419ad8\") " Oct 31 03:17:47.735782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735702 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir" pod "9d6bd592-ed4c-4a14-9455-fbdd18419ad8" (UID: "9d6bd592-ed4c-4a14-9455-fbdd18419ad8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.735782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735771 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home" pod "9d6bd592-ed4c-4a14-9455-fbdd18419ad8" (UID: "9d6bd592-ed4c-4a14-9455-fbdd18419ad8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.736058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.735991 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache" pod "9d6bd592-ed4c-4a14-9455-fbdd18419ad8" (UID: "9d6bd592-ed4c-4a14-9455-fbdd18419ad8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.736404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.736385 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.736477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.736405 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.736477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.736419 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.738075 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.738053 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs" pod "9d6bd592-ed4c-4a14-9455-fbdd18419ad8" (UID: "9d6bd592-ed4c-4a14-9455-fbdd18419ad8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:47.738377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.738360 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm" pod "9d6bd592-ed4c-4a14-9455-fbdd18419ad8" (UID: "9d6bd592-ed4c-4a14-9455-fbdd18419ad8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.738427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.738413 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l" pod "9d6bd592-ed4c-4a14-9455-fbdd18419ad8" (UID: "9d6bd592-ed4c-4a14-9455-fbdd18419ad8"). InnerVolumeSpecName "kube-api-access-rvn7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:47.838539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.838508 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvn7l\" (UniqueName: \"kubernetes.io/projected/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-kube-api-access-rvn7l\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.838539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.838537 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.838539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:47.838548 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6bd592-ed4c-4a14-9455-fbdd18419ad8-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:48.113041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.113003 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" podUID="04758a9b-21b7-443c-93c9-fb092dcbc771" containerName="payload-processing" containerID="cri-o://6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5" gracePeriod=25 Oct 31 03:17:48.558762 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.558729 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:17:48.564072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.564020 4854 generic.go:356] "Generic (PLEG): container finished" podID="04758a9b-21b7-443c-93c9-fb092dcbc771" containerID="6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5" exitCode=0 Oct 31 03:17:48.564316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.564177 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" Oct 31 03:17:48.565773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.565689 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" event={"ID":"04758a9b-21b7-443c-93c9-fb092dcbc771","Type":"ContainerDied","Data":"6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5"} Oct 31 03:17:48.565773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.565742 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx" event={"ID":"04758a9b-21b7-443c-93c9-fb092dcbc771","Type":"ContainerDied","Data":"f97c4ba233c8fcb56267ef4332cbaff93758c179357e3e73a09a9c73f354ce25"} Oct 31 03:17:48.565773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.565773 4854 scope.go:122] "RemoveContainer" containerID="6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5" Oct 31 03:17:48.568669 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.568562 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" Oct 31 03:17:48.568912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.568887 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m" event={"ID":"9d6bd592-ed4c-4a14-9455-fbdd18419ad8","Type":"ContainerDied","Data":"12ef4df9f1e47c912e883e1380b7f8108143fba87296584d04c3da423d04f2a5"} Oct 31 03:17:48.588875 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.588849 4854 scope.go:122] "RemoveContainer" containerID="6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5" Oct 31 03:17:48.590992 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:17:48.590956 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5\": container with ID starting with 6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5 not found: ID does not exist" containerID="6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5" Oct 31 03:17:48.591095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.591004 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5"} err="failed to get container status \"6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5\": rpc error: code = NotFound desc = could not find container \"6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5\": container with ID starting with 6b3265b292968aa83c306ad2bfa40076e214d478146e594249f07e14805d6fd5 not found: ID does not exist" Oct 31 03:17:48.591095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.591029 4854 scope.go:122] "RemoveContainer" containerID="a3beddf22dfee12a7dacafff7bbaf19b3165b4b35acb7612f7c072a0e4422aab" Oct 31 03:17:48.611318 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.610175 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m"] Oct 31 03:17:48.626159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.626094 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/premium-w5-56afbd-kserve-5468b7768c-s7j4m"] Oct 31 03:17:48.686717 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.686694 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume\") pod \"04758a9b-21b7-443c-93c9-fb092dcbc771\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " Oct 31 03:17:48.687061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.686796 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq\" (UniqueName: \"kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq\") pod \"04758a9b-21b7-443c-93c9-fb092dcbc771\" (UID: \"04758a9b-21b7-443c-93c9-fb092dcbc771\") " Oct 31 03:17:48.687112 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.687081 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume" pod "04758a9b-21b7-443c-93c9-fb092dcbc771" (UID: "04758a9b-21b7-443c-93c9-fb092dcbc771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:48.687643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.687623 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04758a9b-21b7-443c-93c9-fb092dcbc771-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:48.709126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.709088 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq" pod "04758a9b-21b7-443c-93c9-fb092dcbc771" (UID: "04758a9b-21b7-443c-93c9-fb092dcbc771"). InnerVolumeSpecName "kube-api-access-94wlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:48.789804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.789771 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wlq\" (UniqueName: \"kubernetes.io/projected/04758a9b-21b7-443c-93c9-fb092dcbc771-kube-api-access-94wlq\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:48.889726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.889652 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx"] Oct 31 03:17:48.895524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:48.895498 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-750c955f-6d5fbf8ff-tw7hx"] Oct 31 03:17:50.384274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:50.376361 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:17:50.626346 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:50.599666 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04758a9b-21b7-443c-93c9-fb092dcbc771" path="/var/lib/kubelet/pods/04758a9b-21b7-443c-93c9-fb092dcbc771/volumes" Oct 31 03:17:50.627788 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:50.627756 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" path="/var/lib/kubelet/pods/9d6bd592-ed4c-4a14-9455-fbdd18419ad8/volumes" Oct 31 03:17:52.487294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:52.484591 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:17:57.817937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:57.817902 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl"] Oct 31 03:17:57.818440 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:57.818155 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" podUID="605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" containerName="maas-api" containerID="cri-o://43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb" gracePeriod=30 Oct 31 03:17:58.619359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.619327 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:17:58.717914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.717671 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf\" (UniqueName: \"kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf\") pod \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " Oct 31 03:17:58.717914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.717744 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\") pod \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " Oct 31 03:17:58.717914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.717826 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca\") pod \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " Oct 31 03:17:58.719433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.718510 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs\") pod \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\" (UID: \"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b\") " Oct 31 03:17:58.720879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.720849 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca" pod "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" (UID: "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:58.768361 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.757101 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs" pod "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" (UID: "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:58.769172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.768837 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf" pod "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" (UID: "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b"). InnerVolumeSpecName "kube-api-access-rhclf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:58.808019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.807709 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls" pod "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" (UID: "605d3f71-b3b6-442f-b896-c6b6d1c6ed2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:58.822651 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.822624 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhclf\" (UniqueName: \"kubernetes.io/projected/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-kube-api-access-rhclf\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.822651 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.822652 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.823092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.822664 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.823092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.822677 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.901818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.901783 4854 generic.go:356] "Generic (PLEG): container finished" podID="605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" containerID="43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb" exitCode=0 Oct 31 03:17:58.901960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.901841 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" Oct 31 03:17:58.901960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.901861 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" event={"ID":"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b","Type":"ContainerDied","Data":"43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb"} Oct 31 03:17:58.901960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.901917 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl" event={"ID":"605d3f71-b3b6-442f-b896-c6b6d1c6ed2b","Type":"ContainerDied","Data":"701996fa0dceb5d16724544d8bfeecc1b3e2d6d0dc7d98f5971cdc3ae17dbf7d"} Oct 31 03:17:58.901960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.901941 4854 scope.go:122] "RemoveContainer" containerID="43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb" Oct 31 03:17:58.914886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.914863 4854 scope.go:122] "RemoveContainer" containerID="43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb" Oct 31 03:17:58.915614 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:17:58.915585 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb\": container with ID starting with 43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb not found: ID does not exist" containerID="43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb" Oct 31 03:17:58.915750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.915621 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb"} err="failed to get container status \"43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb\": rpc error: code = NotFound desc = could not find container \"43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb\": container with ID starting with 43687ccf1919571e03df3046556ad12b01aafdbef1bca8d6f254385f8e0cf9fb not found: ID does not exist" Oct 31 03:17:58.929640 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.929617 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl"] Oct 31 03:17:58.935872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:17:58.935851 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-750c955f-6d65db6b58-vb8pl"] Oct 31 03:18:00.351164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:00.351127 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:18:00.459483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:00.459443 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" path="/var/lib/kubelet/pods/605d3f71-b3b6-442f-b896-c6b6d1c6ed2b/volumes" Oct 31 03:18:02.412707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.412680 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:18:02.520425 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.520397 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w\" (UniqueName: \"kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w\") pod \"805261a2-7bf7-4b7e-8d55-28c65c030d58\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " Oct 31 03:18:02.520616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.520487 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume\") pod \"805261a2-7bf7-4b7e-8d55-28c65c030d58\" (UID: \"805261a2-7bf7-4b7e-8d55-28c65c030d58\") " Oct 31 03:18:02.520842 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.520820 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume" pod "805261a2-7bf7-4b7e-8d55-28c65c030d58" (UID: "805261a2-7bf7-4b7e-8d55-28c65c030d58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:02.521197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.521177 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805261a2-7bf7-4b7e-8d55-28c65c030d58-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:02.522673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.522651 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w" pod "805261a2-7bf7-4b7e-8d55-28c65c030d58" (UID: "805261a2-7bf7-4b7e-8d55-28c65c030d58"). InnerVolumeSpecName "kube-api-access-25v7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:02.623168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:02.623140 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25v7w\" (UniqueName: \"kubernetes.io/projected/805261a2-7bf7-4b7e-8d55-28c65c030d58-kube-api-access-25v7w\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:03.093464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.093430 4854 generic.go:356] "Generic (PLEG): container finished" podID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerID="9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726" exitCode=0 Oct 31 03:18:03.093682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.093489 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" Oct 31 03:18:03.093682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.093513 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" event={"ID":"805261a2-7bf7-4b7e-8d55-28c65c030d58","Type":"ContainerDied","Data":"9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726"} Oct 31 03:18:03.093682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.093581 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r" event={"ID":"805261a2-7bf7-4b7e-8d55-28c65c030d58","Type":"ContainerDied","Data":"ec5c8645c96c4bbe6ae5aa7dd22c4c85ebcb6944c6e98b12b96c376fb041e4e3"} Oct 31 03:18:03.093682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.093611 4854 scope.go:122] "RemoveContainer" containerID="9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726" Oct 31 03:18:03.110276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.110148 4854 scope.go:122] "RemoveContainer" containerID="9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726" Oct 31 03:18:03.111448 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:03.110700 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726\": container with ID starting with 9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726 not found: ID does not exist" containerID="9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726" Oct 31 03:18:03.111448 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.110739 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726"} err="failed to get container status \"9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726\": rpc error: code = NotFound desc = could not find container \"9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726\": container with ID starting with 9bae8a17cba80607cd934bd3261ccfabdb0b14c92012706a13afd87f2f3c9726 not found: ID does not exist" Oct 31 03:18:03.139264 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.139218 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r"] Oct 31 03:18:03.143333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:03.143313 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-56afbd-9d894dbc7-6rt9r"] Oct 31 03:18:04.459463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:04.459431 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" path="/var/lib/kubelet/pods/805261a2-7bf7-4b7e-8d55-28c65c030d58/volumes" Oct 31 03:18:07.090271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:07.088816 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h"] Oct 31 03:18:07.090271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:07.089097 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" containerID="cri-o://188d359394a320293e0deb007e4d7372627e98a307716f9aa893b5b12dca5a2a" gracePeriod=30 Oct 31 03:18:08.617275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:08.615469 4854 patch_prober.go:29] interesting pod/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:18:08.617275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:08.615538 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:18:09.251762 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:09.251725 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:18:12.820133 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.820064 4854 generic.go:356] "Generic (PLEG): container finished" podID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerID="188d359394a320293e0deb007e4d7372627e98a307716f9aa893b5b12dca5a2a" exitCode=0 Oct 31 03:18:12.820588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.820145 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" event={"ID":"8621502b-46f5-44a8-8bf2-eccd181a7ee8","Type":"ContainerDied","Data":"188d359394a320293e0deb007e4d7372627e98a307716f9aa893b5b12dca5a2a"} Oct 31 03:18:12.820588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.820209 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" event={"ID":"8621502b-46f5-44a8-8bf2-eccd181a7ee8","Type":"ContainerDied","Data":"3c0c24530b91f543f7b49fb99aacc9c4826cc6ee040f41c8325d805fdd693295"} Oct 31 03:18:12.820588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.820228 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c0c24530b91f543f7b49fb99aacc9c4826cc6ee040f41c8325d805fdd693295" Oct 31 03:18:12.820588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.820241 4854 scope.go:122] "RemoveContainer" containerID="24fbc9a25c1ac5ce9e21484bcf35b394dc48ee5540228de635184f7b6e9fa146" Oct 31 03:18:12.837502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.837482 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:18:12.910600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.910566 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.910831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.910688 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.910831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.910773 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.910831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.910821 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.911044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.910909 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.911044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.910957 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.911044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.911033 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.911200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.911075 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.911200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.911168 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy\") pod \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\" (UID: \"8621502b-46f5-44a8-8bf2-eccd181a7ee8\") " Oct 31 03:18:12.911200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.911175 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:12.911480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.911428 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:12.911665 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.911513 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:12.912068 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.912043 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:12.912150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.912073 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.912150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.912098 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.912150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.912113 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.912385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.912364 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:12.914796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.914765 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:12.914952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.914928 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:12.915411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.915387 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:12.915762 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:12.915743 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt" pod "8621502b-46f5-44a8-8bf2-eccd181a7ee8" (UID: "8621502b-46f5-44a8-8bf2-eccd181a7ee8"). InnerVolumeSpecName "kube-api-access-hwxkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:13.013471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.013438 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.013471 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.013467 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.013614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.013480 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.013614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.013492 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.013614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.013504 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxkt\" (UniqueName: \"kubernetes.io/projected/8621502b-46f5-44a8-8bf2-eccd181a7ee8-kube-api-access-hwxkt\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.013614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.013517 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8621502b-46f5-44a8-8bf2-eccd181a7ee8-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:13.957327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.956275 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h" Oct 31 03:18:13.959807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.959027 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8"] Oct 31 03:18:13.959807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.959240 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" containerID="cri-o://4a225f5387ca2836a8d973b4c828d127be6c7b3f0e227fdbcf2fa5493a8c95fc" gracePeriod=30 Oct 31 03:18:13.970390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.970365 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2"] Oct 31 03:18:13.970577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:13.970554 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" containerID="cri-o://c8be71f06e33fdb71d56f1d618acbd60eaa31abefef4f486f12f1413d6b23a32" gracePeriod=30 Oct 31 03:18:14.005305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.005279 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h"] Oct 31 03:18:14.010588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.010565 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-750c955f-openshift-default-7db848f469-hgh7h"] Oct 31 03:18:14.459712 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.459680 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" path="/var/lib/kubelet/pods/8621502b-46f5-44a8-8bf2-eccd181a7ee8/volumes" Oct 31 03:18:14.678147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678118 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk"] Oct 31 03:18:14.678900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678883 4854 cpu_manager.go:414] "containerMap: removing container" podUID="04758a9b-21b7-443c-93c9-fb092dcbc771" containerName="payload-processing" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678902 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04758a9b-21b7-443c-93c9-fb092dcbc771" containerName="payload-processing" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678915 4854 cpu_manager.go:414] "containerMap: removing container" podUID="605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" containerName="maas-api" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678920 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" containerName="maas-api" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678935 4854 cpu_manager.go:414] "containerMap: removing container" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerName="payload-processing" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678940 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerName="payload-processing" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678954 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" Oct 31 03:18:14.678967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678958 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678981 4854 cpu_manager.go:414] "containerMap: removing container" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.678986 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679003 4854 cpu_manager.go:414] "containerMap: removing container" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679007 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679120 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="04758a9b-21b7-443c-93c9-fb092dcbc771" containerName="payload-processing" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679130 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="605d3f71-b3b6-442f-b896-c6b6d1c6ed2b" containerName="maas-api" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679137 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="805261a2-7bf7-4b7e-8d55-28c65c030d58" containerName="payload-processing" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679144 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6bd592-ed4c-4a14-9455-fbdd18419ad8" containerName="main" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679152 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" Oct 31 03:18:14.679177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.679161 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="8621502b-46f5-44a8-8bf2-eccd181a7ee8" containerName="istio-proxy" Oct 31 03:18:14.684208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.684181 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:14.699272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.693338 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs"] Oct 31 03:18:14.700083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.700058 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk"] Oct 31 03:18:14.700223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.700168 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:14.723895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.706634 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1fd580-dockercfg-fr5n8\"" Oct 31 03:18:14.723895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.706770 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1fd580\"" Oct 31 03:18:14.723895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.713919 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs"] Oct 31 03:18:14.758434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.758396 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:14.758596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.758473 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrvt\" (UniqueName: \"kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt\") pod \"payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:14.758596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.758531 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:14.758721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.758620 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdg7\" (UniqueName: \"kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7\") pod \"payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:14.973380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.973341 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:14.974009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.973949 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrvt\" (UniqueName: \"kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt\") pod \"payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:14.984441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.980380 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:14.984648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.984629 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:14.985816 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.985657 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:14.992342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:14.992313 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdg7\" (UniqueName: \"kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7\") pod \"payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:15.143998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:15.143842 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdg7\" (UniqueName: \"kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7\") pod \"payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:15.216274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:15.212905 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrvt\" (UniqueName: \"kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt\") pod \"payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:15.356206 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:15.355604 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:15.357291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:15.357236 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:15.722653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:15.722599 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk"] Oct 31 03:18:15.722848 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:15.722824 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40de8418_e312_44f2_b81b_b489af17a8b3.slice/crio-4f6d5f108d17c8a13af938939dd0128ec705ef459bb5376b8efb78fb293e8d0f WatchSource:0}: Error finding container 4f6d5f108d17c8a13af938939dd0128ec705ef459bb5376b8efb78fb293e8d0f: Status 404 returned error can't find the container with id 4f6d5f108d17c8a13af938939dd0128ec705ef459bb5376b8efb78fb293e8d0f Oct 31 03:18:15.725118 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:15.725087 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fd4cad_6df0_4c1c_aa28_c2b0a34c40bb.slice/crio-1abab4fbc34222f2a16ea4625565c13efce982b07881bce660233d02d31b37c2 WatchSource:0}: Error finding container 1abab4fbc34222f2a16ea4625565c13efce982b07881bce660233d02d31b37c2: Status 404 returned error can't find the container with id 1abab4fbc34222f2a16ea4625565c13efce982b07881bce660233d02d31b37c2 Oct 31 03:18:15.727334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:15.727315 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs"] Oct 31 03:18:16.140994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.140960 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" event={"ID":"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb","Type":"ContainerStarted","Data":"d5680159cf5cbb661c9e415a4b433caf25a5c5825254f3e8fce98908b5c63936"} Oct 31 03:18:16.141473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.141014 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" event={"ID":"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb","Type":"ContainerStarted","Data":"1abab4fbc34222f2a16ea4625565c13efce982b07881bce660233d02d31b37c2"} Oct 31 03:18:16.141473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.141041 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:16.142790 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.142765 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" event={"ID":"40de8418-e312-44f2-b81b-b489af17a8b3","Type":"ContainerStarted","Data":"88877cc0bfb14263b678687223cff51d28f6c1c4553689376588074bd5edb976"} Oct 31 03:18:16.142912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.142805 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" event={"ID":"40de8418-e312-44f2-b81b-b489af17a8b3","Type":"ContainerStarted","Data":"4f6d5f108d17c8a13af938939dd0128ec705ef459bb5376b8efb78fb293e8d0f"} Oct 31 03:18:16.142912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.142838 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:16.158044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.157962 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" podStartSLOduration=2.157946099 podStartE2EDuration="2.157946099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:16.157484501 +0000 UTC m=+1379.854651829" watchObservedRunningTime="2026-10-31 03:18:16.157946099 +0000 UTC m=+1379.855113406" Oct 31 03:18:16.176612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:16.176566 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" podStartSLOduration=2.176554588 podStartE2EDuration="2.176554588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:16.174378842 +0000 UTC m=+1379.871546159" watchObservedRunningTime="2026-10-31 03:18:16.176554588 +0000 UTC m=+1379.873721892" Oct 31 03:18:19.326618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.325572 4854 generic.go:356] "Generic (PLEG): container finished" podID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerID="4a225f5387ca2836a8d973b4c828d127be6c7b3f0e227fdbcf2fa5493a8c95fc" exitCode=0 Oct 31 03:18:19.326618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.325709 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" event={"ID":"a8122523-cf19-4951-8e8c-54dec0ee3aca","Type":"ContainerDied","Data":"4a225f5387ca2836a8d973b4c828d127be6c7b3f0e227fdbcf2fa5493a8c95fc"} Oct 31 03:18:19.327606 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.327425 4854 generic.go:356] "Generic (PLEG): container finished" podID="b01173b7-af83-4b2d-9970-80534aad8f63" containerID="c8be71f06e33fdb71d56f1d618acbd60eaa31abefef4f486f12f1413d6b23a32" exitCode=0 Oct 31 03:18:19.327606 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.327609 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" event={"ID":"b01173b7-af83-4b2d-9970-80534aad8f63","Type":"ContainerDied","Data":"c8be71f06e33fdb71d56f1d618acbd60eaa31abefef4f486f12f1413d6b23a32"} Oct 31 03:18:19.378041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.378017 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:18:19.464497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.464463 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:18:19.515027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.514994 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.515295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515240 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.515429 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515396 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.515504 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515434 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.515565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515517 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.515565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515548 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.515672 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515616 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.516185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515775 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.516185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515982 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.516185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.515982 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.516185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516066 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.516185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516105 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.516653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516609 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.516745 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516685 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.516745 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516729 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.516859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516782 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.517967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516909 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.517967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.516952 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token\") pod \"a8122523-cf19-4951-8e8c-54dec0ee3aca\" (UID: \"a8122523-cf19-4951-8e8c-54dec0ee3aca\") " Oct 31 03:18:19.518194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.518171 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.518194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.518199 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.518434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.518213 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.518783 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.518757 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "kube-api-access-zttf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.518872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.518778 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.519031 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.519015 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:19.519320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.519299 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.519840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.519811 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:19.520511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.520486 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.520635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.520610 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:19.521722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.521696 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.521845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.521745 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "kube-api-access-cwfpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.522115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.522094 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.523386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.523350 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.523756 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.523733 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo" pod "a8122523-cf19-4951-8e8c-54dec0ee3aca" (UID: "a8122523-cf19-4951-8e8c-54dec0ee3aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:19.618924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.618847 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.618924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.618889 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.619176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.618953 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs\") pod \"b01173b7-af83-4b2d-9970-80534aad8f63\" (UID: \"b01173b7-af83-4b2d-9970-80534aad8f63\") " Oct 31 03:18:19.619176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619154 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.619391 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619368 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.619467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619443 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs" pod "b01173b7-af83-4b2d-9970-80534aad8f63" (UID: "b01173b7-af83-4b2d-9970-80534aad8f63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.619839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619815 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619840 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619875 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8122523-cf19-4951-8e8c-54dec0ee3aca-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619884 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619893 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619908 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619919 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttf4\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-kube-api-access-zttf4\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619929 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619938 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619946 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.619949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619953 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01173b7-af83-4b2d-9970-80534aad8f63-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.620310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619961 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01173b7-af83-4b2d-9970-80534aad8f63-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.620310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619971 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwfpn\" (UniqueName: \"kubernetes.io/projected/b01173b7-af83-4b2d-9970-80534aad8f63-kube-api-access-cwfpn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.620310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619985 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01173b7-af83-4b2d-9970-80534aad8f63-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.620310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:19.619996 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8122523-cf19-4951-8e8c-54dec0ee3aca-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:20.415012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.414924 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" Oct 31 03:18:20.415012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.414954 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8" event={"ID":"a8122523-cf19-4951-8e8c-54dec0ee3aca","Type":"ContainerDied","Data":"59e1018217146a0bca55484d55a99eb9c1e09d52328d63b11ab3146cb5d5f47a"} Oct 31 03:18:20.415628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.415025 4854 scope.go:122] "RemoveContainer" containerID="4a225f5387ca2836a8d973b4c828d127be6c7b3f0e227fdbcf2fa5493a8c95fc" Oct 31 03:18:20.418299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.416823 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" Oct 31 03:18:20.418299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.416856 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2" event={"ID":"b01173b7-af83-4b2d-9970-80534aad8f63","Type":"ContainerDied","Data":"64f51ba5e9135932ceda1cc6ac6baeb6dee3148516280b672dd1f7bec6a40cc5"} Oct 31 03:18:20.430307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.430285 4854 scope.go:122] "RemoveContainer" containerID="c8be71f06e33fdb71d56f1d618acbd60eaa31abefef4f486f12f1413d6b23a32" Oct 31 03:18:20.446192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.446165 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8"] Oct 31 03:18:20.459355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.459325 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-5bcc8"] Oct 31 03:18:20.466231 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.466207 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2"] Oct 31 03:18:20.467938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:20.467919 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-nc8d2"] Oct 31 03:18:22.322276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.304940 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx"] Oct 31 03:18:22.322276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.305201 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" containerID="cri-o://3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f" gracePeriod=30 Oct 31 03:18:22.367276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.333023 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm"] Oct 31 03:18:22.367276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.333307 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" containerID="cri-o://1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345" gracePeriod=30 Oct 31 03:18:22.446283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.444318 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq"] Oct 31 03:18:22.446283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.444556 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" containerID="cri-o://ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2" gracePeriod=30 Oct 31 03:18:22.807563 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:22.807519 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:23.071333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.069207 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" path="/var/lib/kubelet/pods/a8122523-cf19-4951-8e8c-54dec0ee3aca/volumes" Oct 31 03:18:23.071333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.070012 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" path="/var/lib/kubelet/pods/b01173b7-af83-4b2d-9970-80534aad8f63/volumes" Oct 31 03:18:23.152948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.152417 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:23.762275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.759330 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:23.827274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.814351 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:18:23.827274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.814900 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:18:23.867371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:23.867329 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:18:24.047330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.042825 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:18:24.228395 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.227546 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:18:24.229172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.229147 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:18:24.308330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.304146 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:18:24.551012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.550964 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:24.833386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.830378 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:24.844743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.844716 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:18:24.879053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:24.877752 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:25.037272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.037020 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:18:25.093444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.093348 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:18:25.119191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.119163 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:18:25.184196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.182803 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:18:25.236454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.236343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:25.264377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.264347 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:18:25.264645 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:25.264627 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:18:26.208277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:26.205328 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:18:26.208277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:26.205393 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:18:27.154327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.154300 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:18:27.154519 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.154430 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:18:27.466414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.466380 4854 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:18:27.466857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.466449 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:18:27.647862 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.647828 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:18:27.722648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.722551 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.722841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.722656 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.722841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.722706 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.722983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.722915 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.723042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.722972 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.723042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723016 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723269 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723311 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723347 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723400 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket\") pod \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\" (UID: \"fc78da3f-c4c7-4ff0-b398-388d0a07a04f\") " Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723528 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723671 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723701 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.723878 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.724548 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.724572 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.724587 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.724601 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.724655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.724615 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.726566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.726318 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.728338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.728275 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "kube-api-access-8l27m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.728338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.728311 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.728586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.728562 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo" pod "fc78da3f-c4c7-4ff0-b398-388d0a07a04f" (UID: "fc78da3f-c4c7-4ff0-b398-388d0a07a04f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:27.850222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.850181 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l27m\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-kube-api-access-8l27m\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.850222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.850218 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.850853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.850234 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.850853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.850264 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc78da3f-c4c7-4ff0-b398-388d0a07a04f-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.856373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.856217 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:18:27.950799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.950768 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.950825 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951001 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.950940 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951004 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951066 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951094 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951110 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951130 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.951435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951151 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert\") pod \"b898ea0f-f027-4065-bab3-cedbab0d2975\" (UID: \"b898ea0f-f027-4065-bab3-cedbab0d2975\") " Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951786 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951799 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.951906 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.952360 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.952452 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.952479 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.952592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.952494 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.953383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.953361 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.955686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.955660 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.956955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.956933 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "kube-api-access-c2bln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.960302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.960202 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:27.961642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:27.961619 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy" pod "b898ea0f-f027-4065-bab3-cedbab0d2975" (UID: "b898ea0f-f027-4065-bab3-cedbab0d2975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:28.054150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.054129 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.054150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.054151 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.054311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.054160 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.054311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.054169 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.054311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.054178 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2bln\" (UniqueName: \"kubernetes.io/projected/b898ea0f-f027-4065-bab3-cedbab0d2975-kube-api-access-c2bln\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.054311 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.054187 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b898ea0f-f027-4065-bab3-cedbab0d2975-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.350042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.349957 4854 generic.go:356] "Generic (PLEG): container finished" podID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerID="ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2" exitCode=0 Oct 31 03:18:28.350042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.350003 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" Oct 31 03:18:28.350042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.350032 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" event={"ID":"b898ea0f-f027-4065-bab3-cedbab0d2975","Type":"ContainerDied","Data":"ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2"} Oct 31 03:18:28.350307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.350078 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq" event={"ID":"b898ea0f-f027-4065-bab3-cedbab0d2975","Type":"ContainerDied","Data":"400629ca0e50982efb2a47869f28767cbfb5b4d06d399b625c31b9f6a330c6a2"} Oct 31 03:18:28.350307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.350100 4854 scope.go:122] "RemoveContainer" containerID="ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2" Oct 31 03:18:28.352323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.352284 4854 generic.go:356] "Generic (PLEG): container finished" podID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerID="1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345" exitCode=0 Oct 31 03:18:28.352323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.352302 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" Oct 31 03:18:28.352323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.352299 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" event={"ID":"fc78da3f-c4c7-4ff0-b398-388d0a07a04f","Type":"ContainerDied","Data":"1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345"} Oct 31 03:18:28.352571 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.352363 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm" event={"ID":"fc78da3f-c4c7-4ff0-b398-388d0a07a04f","Type":"ContainerDied","Data":"c4556cee2e8a3ea049e6bdac34def918b5a3aa6e4e00673977e0c0e7eeab0340"} Oct 31 03:18:28.362122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.362104 4854 scope.go:122] "RemoveContainer" containerID="ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2" Oct 31 03:18:28.362437 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:28.362414 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2\": container with ID starting with ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2 not found: ID does not exist" containerID="ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2" Oct 31 03:18:28.362510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.362448 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2"} err="failed to get container status \"ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2\": rpc error: code = NotFound desc = could not find container \"ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2\": container with ID starting with ffac70e9c70af1e6671657a16c3cea4812ed48467961fa1b5c5d377c07af7bf2 not found: ID does not exist" Oct 31 03:18:28.362510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.362466 4854 scope.go:122] "RemoveContainer" containerID="1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345" Oct 31 03:18:28.374266 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.374231 4854 scope.go:122] "RemoveContainer" containerID="ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af" Oct 31 03:18:28.379212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.379186 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq"] Oct 31 03:18:28.382592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.382573 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-trfnq"] Oct 31 03:18:28.404380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.404353 4854 scope.go:122] "RemoveContainer" containerID="1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345" Oct 31 03:18:28.404853 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:28.404811 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345\": container with ID starting with 1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345 not found: ID does not exist" containerID="1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345" Oct 31 03:18:28.405035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.405006 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345"} err="failed to get container status \"1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345\": rpc error: code = NotFound desc = could not find container \"1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345\": container with ID starting with 1f6efb7da01d7c45638932056989fc22298c66d942ca4a15e15f21e2a5d20345 not found: ID does not exist" Oct 31 03:18:28.405141 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.405127 4854 scope.go:122] "RemoveContainer" containerID="ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af" Oct 31 03:18:28.405363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.405348 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm"] Oct 31 03:18:28.405587 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:28.405565 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af\": container with ID starting with ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af not found: ID does not exist" containerID="ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af" Oct 31 03:18:28.405709 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.405688 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af"} err="failed to get container status \"ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af\": rpc error: code = NotFound desc = could not find container \"ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af\": container with ID starting with ba1351787cdf2c3d7c4d0ca2ccd6ad9e0f2aee5bd7e8c49b66ce7394e42ef0af not found: ID does not exist" Oct 31 03:18:28.410038 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.410017 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-xg9sm"] Oct 31 03:18:28.459497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.459464 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" path="/var/lib/kubelet/pods/b898ea0f-f027-4065-bab3-cedbab0d2975/volumes" Oct 31 03:18:28.459931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:28.459918 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" path="/var/lib/kubelet/pods/fc78da3f-c4c7-4ff0-b398-388d0a07a04f/volumes" Oct 31 03:18:29.960518 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:29.960494 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:18:30.088207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088176 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088227 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088285 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088327 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088409 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088457 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088574 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:30.088751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088665 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088724 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:30.088857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088759 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:30.088857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088774 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.088857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.088841 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs\") pod \"76360468-ef71-4e93-b649-6e936156ee06\" (UID: \"76360468-ef71-4e93-b649-6e936156ee06\") " Oct 31 03:18:30.089042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089023 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:30.089101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089047 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:30.089506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089485 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.089506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089506 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.089657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089517 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.089657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089526 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76360468-ef71-4e93-b649-6e936156ee06-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.089657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.089534 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.091014 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.090991 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:30.091014 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.091003 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "kube-api-access-222mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:30.091325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.091291 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:30.091444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.091325 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token" pod "76360468-ef71-4e93-b649-6e936156ee06" (UID: "76360468-ef71-4e93-b649-6e936156ee06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:30.191333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.191298 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-222mg\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-kube-api-access-222mg\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.191333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.191331 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76360468-ef71-4e93-b649-6e936156ee06-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.191522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.191342 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76360468-ef71-4e93-b649-6e936156ee06-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.191522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.191351 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76360468-ef71-4e93-b649-6e936156ee06-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:30.388872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.388780 4854 generic.go:356] "Generic (PLEG): container finished" podID="76360468-ef71-4e93-b649-6e936156ee06" containerID="3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f" exitCode=0 Oct 31 03:18:30.388872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.388831 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" Oct 31 03:18:30.388872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.388832 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" event={"ID":"76360468-ef71-4e93-b649-6e936156ee06","Type":"ContainerDied","Data":"3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f"} Oct 31 03:18:30.389135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.388906 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx" event={"ID":"76360468-ef71-4e93-b649-6e936156ee06","Type":"ContainerDied","Data":"5ed6576ce4f5cdc6aceb6a4545d66658fef72196b48a7d5f28834741aa2f2833"} Oct 31 03:18:30.389135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.388935 4854 scope.go:122] "RemoveContainer" containerID="3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f" Oct 31 03:18:30.400305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.400244 4854 scope.go:122] "RemoveContainer" containerID="3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f" Oct 31 03:18:30.400601 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:30.400574 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f\": container with ID starting with 3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f not found: ID does not exist" containerID="3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f" Oct 31 03:18:30.400661 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.400612 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f"} err="failed to get container status \"3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f\": rpc error: code = NotFound desc = could not find container \"3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f\": container with ID starting with 3f2ebaf3a71687ba0769c04b0ebd37a1ef902f3f7264c0f0c83ade246689c66f not found: ID does not exist" Oct 31 03:18:30.412887 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.412856 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx"] Oct 31 03:18:30.418150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.418126 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-82mpx"] Oct 31 03:18:30.458837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:30.458808 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76360468-ef71-4e93-b649-6e936156ee06" path="/var/lib/kubelet/pods/76360468-ef71-4e93-b649-6e936156ee06/volumes" Oct 31 03:18:34.476674 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:34.476638 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt"] Oct 31 03:18:34.477386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:34.476885 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" containerID="cri-o://7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e" gracePeriod=30 Oct 31 03:18:35.849495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.849458 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf"] Oct 31 03:18:35.850722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850695 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" Oct 31 03:18:35.850722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850721 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850735 4854 cpu_manager.go:414] "containerMap: removing container" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850743 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850773 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850781 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850797 4854 cpu_manager.go:414] "containerMap: removing container" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850804 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850845 4854 cpu_manager.go:414] "containerMap: removing container" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850852 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850897 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" Oct 31 03:18:35.850931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.850905 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" Oct 31 03:18:35.851467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.851075 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8122523-cf19-4951-8e8c-54dec0ee3aca" containerName="istio-proxy" Oct 31 03:18:35.851467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.851094 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" Oct 31 03:18:35.851467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.851340 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01173b7-af83-4b2d-9970-80534aad8f63" containerName="istio-proxy" Oct 31 03:18:35.851467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.851377 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc78da3f-c4c7-4ff0-b398-388d0a07a04f" containerName="istio-proxy" Oct 31 03:18:35.851467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.851388 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="76360468-ef71-4e93-b649-6e936156ee06" containerName="istio-proxy" Oct 31 03:18:35.851467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.851408 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="b898ea0f-f027-4065-bab3-cedbab0d2975" containerName="istio-proxy" Oct 31 03:18:35.856074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.856047 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:35.865871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.865851 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a2bec7-openshift-default-dockercfg-xgswz\"" Oct 31 03:18:35.867445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.867422 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf"] Oct 31 03:18:35.968322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.945972 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:35.968322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.946079 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:35.968322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.946150 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:35.968322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:35.946199 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.005275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.003975 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.005275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.004056 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6g7\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.005275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.004127 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.005275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.004154 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.005275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.004373 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.118684 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.118580 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.118882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.118766 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.118882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.118845 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.118924 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.118982 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119120 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.119055 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119120 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.119105 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6g7\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.119171 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.119204 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.119940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.119911 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.120210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.120165 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.120860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.120836 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.214276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.206966 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.214836 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.214806 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.270562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.270528 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.303273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.302906 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.312043 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.311978 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6g7\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.321281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.320492 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo\") pod \"e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.376505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.369988 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp"] Oct 31 03:18:36.377047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.376987 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" podUID="6a8ffab6-dfb1-4979-8090-d718a41aad1f" containerName="maas-api" containerID="cri-o://de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004" gracePeriod=30 Oct 31 03:18:36.477747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.477717 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:36.988280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.986873 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:36.988280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.986966 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:36.988280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:36.987012 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:36.989516 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:36.989491 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9661d7af_7aba_4d48_bf9a_1f97f804b29a.slice/crio-792c63e1b9afc19c92a128ca6f9766b9c6ac9cf6c509a69aa4cea2f8137292c9 WatchSource:0}: Error finding container 792c63e1b9afc19c92a128ca6f9766b9c6ac9cf6c509a69aa4cea2f8137292c9: Status 404 returned error can't find the container with id 792c63e1b9afc19c92a128ca6f9766b9c6ac9cf6c509a69aa4cea2f8137292c9 Oct 31 03:18:37.036645 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.027834 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf"] Oct 31 03:18:37.198463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.191003 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:18:37.357703 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.357674 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs\") pod \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " Oct 31 03:18:37.357957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.357711 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx\" (UniqueName: \"kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx\") pod \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " Oct 31 03:18:37.358037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.357968 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca\") pod \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " Oct 31 03:18:37.358322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.358300 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca" pod "6a8ffab6-dfb1-4979-8090-d718a41aad1f" (UID: "6a8ffab6-dfb1-4979-8090-d718a41aad1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:37.358408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.358323 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls\") pod \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\" (UID: \"6a8ffab6-dfb1-4979-8090-d718a41aad1f\") " Oct 31 03:18:37.358996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.358972 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a8ffab6-dfb1-4979-8090-d718a41aad1f-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:37.360843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.360788 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls" pod "6a8ffab6-dfb1-4979-8090-d718a41aad1f" (UID: "6a8ffab6-dfb1-4979-8090-d718a41aad1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:37.360843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.360837 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx" pod "6a8ffab6-dfb1-4979-8090-d718a41aad1f" (UID: "6a8ffab6-dfb1-4979-8090-d718a41aad1f"). InnerVolumeSpecName "kube-api-access-7p8xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:37.361017 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.360996 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs" pod "6a8ffab6-dfb1-4979-8090-d718a41aad1f" (UID: "6a8ffab6-dfb1-4979-8090-d718a41aad1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:37.460612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.460523 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:37.460612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.460549 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a8ffab6-dfb1-4979-8090-d718a41aad1f-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:37.460612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.460563 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8xx\" (UniqueName: \"kubernetes.io/projected/6a8ffab6-dfb1-4979-8090-d718a41aad1f-kube-api-access-7p8xx\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:37.461779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.461753 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" event={"ID":"9661d7af-7aba-4d48-bf9a-1f97f804b29a","Type":"ContainerStarted","Data":"714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e"} Oct 31 03:18:37.461904 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.461812 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" event={"ID":"9661d7af-7aba-4d48-bf9a-1f97f804b29a","Type":"ContainerStarted","Data":"792c63e1b9afc19c92a128ca6f9766b9c6ac9cf6c509a69aa4cea2f8137292c9"} Oct 31 03:18:37.463633 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.463521 4854 generic.go:356] "Generic (PLEG): container finished" podID="6a8ffab6-dfb1-4979-8090-d718a41aad1f" containerID="de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004" exitCode=0 Oct 31 03:18:37.463733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.463718 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" event={"ID":"6a8ffab6-dfb1-4979-8090-d718a41aad1f","Type":"ContainerDied","Data":"de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004"} Oct 31 03:18:37.463796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.463738 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" Oct 31 03:18:37.463796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.463750 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp" event={"ID":"6a8ffab6-dfb1-4979-8090-d718a41aad1f","Type":"ContainerDied","Data":"6463e6df2c1abe40003e461ea46161b8998664ef69148d7d71990e29d6504547"} Oct 31 03:18:37.463796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.463771 4854 scope.go:122] "RemoveContainer" containerID="de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004" Oct 31 03:18:37.476392 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.476372 4854 scope.go:122] "RemoveContainer" containerID="de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004" Oct 31 03:18:37.476680 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:37.476653 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004\": container with ID starting with de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004 not found: ID does not exist" containerID="de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004" Oct 31 03:18:37.476776 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.476692 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004"} err="failed to get container status \"de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004\": rpc error: code = NotFound desc = could not find container \"de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004\": container with ID starting with de401cffa8cdd7b386647ec831f6ed45c93488a8c3006def91e24fff1536d004 not found: ID does not exist" Oct 31 03:18:37.478987 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.478958 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:37.493559 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.493512 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podStartSLOduration=2.493499964 podStartE2EDuration="2.493499964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:37.48961726 +0000 UTC m=+1401.186784592" watchObservedRunningTime="2026-10-31 03:18:37.493499964 +0000 UTC m=+1401.190667268" Oct 31 03:18:37.511439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.511415 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp"] Oct 31 03:18:37.517898 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:37.517878 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-hfqpp"] Oct 31 03:18:38.461107 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:38.461074 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8ffab6-dfb1-4979-8090-d718a41aad1f" path="/var/lib/kubelet/pods/6a8ffab6-dfb1-4979-8090-d718a41aad1f/volumes" Oct 31 03:18:38.483655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:38.483626 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:39.253192 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:39.253144 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:18:39.486503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.486468 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:39.487955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.487935 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:18:39.794623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.794595 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.848932 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849062 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849113 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849268 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849422 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849486 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849538 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849610 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.849870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.849729 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo\") pod \"c4bab621-cecb-4604-b67b-128aa77d685b\" (UID: \"c4bab621-cecb-4604-b67b-128aa77d685b\") " Oct 31 03:18:39.855809 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.855453 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:39.857532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.857359 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:39.858009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.857706 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:39.862082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.862050 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:39.862484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.862388 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:39.881439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.862737 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:39.898276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.891626 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "kube-api-access-ttwdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:39.921502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.920442 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:39.938408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.938333 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token" pod "c4bab621-cecb-4604-b67b-128aa77d685b" (UID: "c4bab621-cecb-4604-b67b-128aa77d685b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954020 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954050 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954066 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954079 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954093 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4bab621-cecb-4604-b67b-128aa77d685b-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954107 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954122 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4bab621-cecb-4604-b67b-128aa77d685b-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954138 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttwdw\" (UniqueName: \"kubernetes.io/projected/c4bab621-cecb-4604-b67b-128aa77d685b-kube-api-access-ttwdw\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:39.954994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:39.954151 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4bab621-cecb-4604-b67b-128aa77d685b-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:40.495737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.495702 4854 generic.go:356] "Generic (PLEG): container finished" podID="c4bab621-cecb-4604-b67b-128aa77d685b" containerID="7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e" exitCode=0 Oct 31 03:18:40.496100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.495749 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" Oct 31 03:18:40.496100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.495783 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" event={"ID":"c4bab621-cecb-4604-b67b-128aa77d685b","Type":"ContainerDied","Data":"7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e"} Oct 31 03:18:40.496100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.495838 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt" event={"ID":"c4bab621-cecb-4604-b67b-128aa77d685b","Type":"ContainerDied","Data":"409b1508d6976eac8403e921929d49f20e1009b1440483accd0275f7255131ca"} Oct 31 03:18:40.496100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.495866 4854 scope.go:122] "RemoveContainer" containerID="7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e" Oct 31 03:18:40.516926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.516909 4854 scope.go:122] "RemoveContainer" containerID="c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb" Oct 31 03:18:40.527436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.527418 4854 scope.go:122] "RemoveContainer" containerID="7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e" Oct 31 03:18:40.527704 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:40.527685 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e\": container with ID starting with 7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e not found: ID does not exist" containerID="7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e" Oct 31 03:18:40.527768 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.527711 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e"} err="failed to get container status \"7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e\": rpc error: code = NotFound desc = could not find container \"7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e\": container with ID starting with 7c3daa7c7c6fde1845f019602fd3de11b2e10038212a83aa3ea867bd6d9ed52e not found: ID does not exist" Oct 31 03:18:40.527768 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.527729 4854 scope.go:122] "RemoveContainer" containerID="c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb" Oct 31 03:18:40.527981 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:40.527956 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb\": container with ID starting with c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb not found: ID does not exist" containerID="c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb" Oct 31 03:18:40.528027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.527978 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb"} err="failed to get container status \"c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb\": rpc error: code = NotFound desc = could not find container \"c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb\": container with ID starting with c1a8bf33dfabb38f30e0bb873993a75d2c6046a9549f6f52de6a9a3667b6ebcb not found: ID does not exist" Oct 31 03:18:40.535558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.535535 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt"] Oct 31 03:18:40.540993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:40.540976 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-780abd-openshift-default-9958655f5-hk4kt"] Oct 31 03:18:41.277363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.277333 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8"] Oct 31 03:18:41.278940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.278914 4854 cpu_manager.go:414] "containerMap: removing container" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" Oct 31 03:18:41.278940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.278939 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" Oct 31 03:18:41.279103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.278979 4854 cpu_manager.go:414] "containerMap: removing container" podUID="6a8ffab6-dfb1-4979-8090-d718a41aad1f" containerName="maas-api" Oct 31 03:18:41.279103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.278986 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8ffab6-dfb1-4979-8090-d718a41aad1f" containerName="maas-api" Oct 31 03:18:41.279103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.279031 4854 cpu_manager.go:414] "containerMap: removing container" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" Oct 31 03:18:41.279103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.279038 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" Oct 31 03:18:41.279410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.279240 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" Oct 31 03:18:41.279483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.279471 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8ffab6-dfb1-4979-8090-d718a41aad1f" containerName="maas-api" Oct 31 03:18:41.279535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.279492 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" containerName="istio-proxy" Oct 31 03:18:41.284798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.284772 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.292867 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.290418 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8"] Oct 31 03:18:41.311164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.311128 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a2bec7-dockercfg-78wkw\"" Oct 31 03:18:41.350401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.350375 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a2bec7\"" Oct 31 03:18:41.409277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.376722 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.430276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.409565 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbf7\" (UniqueName: \"kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7\") pod \"payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.546178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.546090 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.546877 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.546852 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbf7\" (UniqueName: \"kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7\") pod \"payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.575276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.563193 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.640737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.640702 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbf7\" (UniqueName: \"kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7\") pod \"payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:41.904455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:41.904377 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:42.079283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:42.079231 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz"] Oct 31 03:18:42.093849 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:42.093819 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68"] Oct 31 03:18:42.292218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:42.292192 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8"] Oct 31 03:18:42.293133 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:42.293108 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4bede3_2df5_4bc8_bda5_edcf1f5ef0bc.slice/crio-aceb6e2a04910d7f3f5d4c893203c57ae2bf5a3c7bdb7acea742b0b8134b2486 WatchSource:0}: Error finding container aceb6e2a04910d7f3f5d4c893203c57ae2bf5a3c7bdb7acea742b0b8134b2486: Status 404 returned error can't find the container with id aceb6e2a04910d7f3f5d4c893203c57ae2bf5a3c7bdb7acea742b0b8134b2486 Oct 31 03:18:42.650273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:42.648315 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bab621-cecb-4604-b67b-128aa77d685b" path="/var/lib/kubelet/pods/c4bab621-cecb-4604-b67b-128aa77d685b/volumes" Oct 31 03:18:42.659807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:42.656789 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" event={"ID":"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc","Type":"ContainerStarted","Data":"aceb6e2a04910d7f3f5d4c893203c57ae2bf5a3c7bdb7acea742b0b8134b2486"} Oct 31 03:18:43.702018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:43.701541 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" event={"ID":"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc","Type":"ContainerStarted","Data":"455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91"} Oct 31 03:18:43.704321 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:43.702846 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:43.724445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:43.724386 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" podStartSLOduration=2.724370817 podStartE2EDuration="2.724370817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:43.720588301 +0000 UTC m=+1407.417755606" watchObservedRunningTime="2026-10-31 03:18:43.724370817 +0000 UTC m=+1407.421538121" Oct 31 03:18:44.266117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:44.266083 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs"] Oct 31 03:18:44.266326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:44.266298 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" containerID="cri-o://43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179" gracePeriod=30 Oct 31 03:18:44.579365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:44.579288 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b"] Oct 31 03:18:44.579536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:44.579514 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" containerID="cri-o://499bf728150767fb640e00565fa151966a74a3f7b73fe8fd806fbc3a318f7f71" gracePeriod=30 Oct 31 03:18:47.080154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.080088 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" podUID="7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" containerName="payload-processing" containerID="cri-o://ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3" gracePeriod=25 Oct 31 03:18:47.094369 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.094334 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" podUID="4db2452b-ccf0-41a2-a16a-a1fde0d1da65" containerName="payload-pre-processing" containerID="cri-o://cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94" gracePeriod=25 Oct 31 03:18:47.100486 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:47.100447 4854 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7776acb8_b5e8_44d7_8d07_e3b0788ab0f9.slice/crio-ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:18:47.100605 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:47.100586 4854 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7776acb8_b5e8_44d7_8d07_e3b0788ab0f9.slice/crio-ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:18:47.402684 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.402655 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:18:47.437989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.437961 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:18:47.482614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.482581 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz\" (UniqueName: \"kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz\") pod \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " Oct 31 03:18:47.482793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.482713 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn\" (UniqueName: \"kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn\") pod \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " Oct 31 03:18:47.482793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.482751 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume\") pod \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\" (UID: \"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9\") " Oct 31 03:18:47.482988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.482961 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume\") pod \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\" (UID: \"4db2452b-ccf0-41a2-a16a-a1fde0d1da65\") " Oct 31 03:18:47.483581 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.483550 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume" pod "4db2452b-ccf0-41a2-a16a-a1fde0d1da65" (UID: "4db2452b-ccf0-41a2-a16a-a1fde0d1da65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:47.483688 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.483664 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume" pod "7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" (UID: "7776acb8-b5e8-44d7-8d07-e3b0788ab0f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:47.485115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.485089 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn" pod "7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" (UID: "7776acb8-b5e8-44d7-8d07-e3b0788ab0f9"). InnerVolumeSpecName "kube-api-access-flqqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:47.485514 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.485497 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz" pod "4db2452b-ccf0-41a2-a16a-a1fde0d1da65" (UID: "4db2452b-ccf0-41a2-a16a-a1fde0d1da65"). InnerVolumeSpecName "kube-api-access-9bgwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:47.584921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.584892 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flqqn\" (UniqueName: \"kubernetes.io/projected/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-kube-api-access-flqqn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:47.584921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.584923 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:47.585118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.584933 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:47.585118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.584943 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgwz\" (UniqueName: \"kubernetes.io/projected/4db2452b-ccf0-41a2-a16a-a1fde0d1da65-kube-api-access-9bgwz\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:47.745458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.745425 4854 generic.go:356] "Generic (PLEG): container finished" podID="7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" containerID="ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3" exitCode=0 Oct 31 03:18:47.745642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.745483 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" Oct 31 03:18:47.745642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.745510 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" event={"ID":"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9","Type":"ContainerDied","Data":"ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3"} Oct 31 03:18:47.745642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.745567 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz" event={"ID":"7776acb8-b5e8-44d7-8d07-e3b0788ab0f9","Type":"ContainerDied","Data":"4afdea19a5b378f3991db085995d5835ac14d1501752bd1ad0bdb306e8f7e753"} Oct 31 03:18:47.745642 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.745597 4854 scope.go:122] "RemoveContainer" containerID="ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3" Oct 31 03:18:47.747197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.747173 4854 generic.go:356] "Generic (PLEG): container finished" podID="4db2452b-ccf0-41a2-a16a-a1fde0d1da65" containerID="cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94" exitCode=0 Oct 31 03:18:47.747345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.747212 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" event={"ID":"4db2452b-ccf0-41a2-a16a-a1fde0d1da65","Type":"ContainerDied","Data":"cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94"} Oct 31 03:18:47.747345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.747228 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" Oct 31 03:18:47.747345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.747273 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68" event={"ID":"4db2452b-ccf0-41a2-a16a-a1fde0d1da65","Type":"ContainerDied","Data":"4541684a89b0228a6c2883c53d449cf30f43f4e9554666eafbbb12a978918166"} Oct 31 03:18:47.757704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.757688 4854 scope.go:122] "RemoveContainer" containerID="ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3" Oct 31 03:18:47.758022 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:47.757993 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3\": container with ID starting with ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3 not found: ID does not exist" containerID="ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3" Oct 31 03:18:47.758148 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.758031 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3"} err="failed to get container status \"ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3\": rpc error: code = NotFound desc = could not find container \"ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3\": container with ID starting with ac1d16079339581277d83837ebe993578761adf668b715f114cb8d7788fefef3 not found: ID does not exist" Oct 31 03:18:47.758148 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.758055 4854 scope.go:122] "RemoveContainer" containerID="cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94" Oct 31 03:18:47.768319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.768296 4854 scope.go:122] "RemoveContainer" containerID="cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94" Oct 31 03:18:47.768604 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:47.768583 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94\": container with ID starting with cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94 not found: ID does not exist" containerID="cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94" Oct 31 03:18:47.768653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.768613 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94"} err="failed to get container status \"cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94\": rpc error: code = NotFound desc = could not find container \"cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94\": container with ID starting with cc9015789cb81a46d59fa8d22f554bc36060ba271ef4b8738ddaa48f74620a94 not found: ID does not exist" Oct 31 03:18:47.775092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.775068 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz"] Oct 31 03:18:47.779988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.779959 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b93a15-587f6bb447-c6lrz"] Oct 31 03:18:47.794944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.793797 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68"] Oct 31 03:18:47.820438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:47.820374 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b93a15-7f6f7f8fcf-7wg68"] Oct 31 03:18:48.459483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:48.459405 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db2452b-ccf0-41a2-a16a-a1fde0d1da65" path="/var/lib/kubelet/pods/4db2452b-ccf0-41a2-a16a-a1fde0d1da65/volumes" Oct 31 03:18:48.459886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:48.459873 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" path="/var/lib/kubelet/pods/7776acb8-b5e8-44d7-8d07-e3b0788ab0f9/volumes" Oct 31 03:18:49.748970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.748943 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:18:49.783303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.782265 4854 generic.go:356] "Generic (PLEG): container finished" podID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerID="499bf728150767fb640e00565fa151966a74a3f7b73fe8fd806fbc3a318f7f71" exitCode=0 Oct 31 03:18:49.783303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.782445 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" event={"ID":"5887b115-7ccf-41e8-b37f-eb5c2a326907","Type":"ContainerDied","Data":"499bf728150767fb640e00565fa151966a74a3f7b73fe8fd806fbc3a318f7f71"} Oct 31 03:18:49.791839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.791702 4854 generic.go:356] "Generic (PLEG): container finished" podID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerID="43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179" exitCode=0 Oct 31 03:18:49.791839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.791805 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" event={"ID":"728dec5f-bdf3-45ff-ab1c-133444da826c","Type":"ContainerDied","Data":"43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179"} Oct 31 03:18:49.791839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.791842 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" event={"ID":"728dec5f-bdf3-45ff-ab1c-133444da826c","Type":"ContainerDied","Data":"9f0d598505d407e4626e1d69ab322b62e3bbfb6e2c4be8740cb582435a0fe0f8"} Oct 31 03:18:49.792074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.791866 4854 scope.go:122] "RemoveContainer" containerID="43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179" Oct 31 03:18:49.792074 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.791990 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs" Oct 31 03:18:49.808996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.806660 4854 scope.go:122] "RemoveContainer" containerID="43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179" Oct 31 03:18:49.808996 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:18:49.807015 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179\": container with ID starting with 43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179 not found: ID does not exist" containerID="43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179" Oct 31 03:18:49.808996 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.807053 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179"} err="failed to get container status \"43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179\": rpc error: code = NotFound desc = could not find container \"43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179\": container with ID starting with 43ad3f23d834de5aab9fb85de096feeb9fb64173cc6c89947f60398b13271179 not found: ID does not exist" Oct 31 03:18:49.821408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.821371 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s"] Oct 31 03:18:49.822175 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822154 4854 cpu_manager.go:414] "containerMap: removing container" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" Oct 31 03:18:49.822310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822177 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" Oct 31 03:18:49.822310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822200 4854 cpu_manager.go:414] "containerMap: removing container" podUID="7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" containerName="payload-processing" Oct 31 03:18:49.822310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822206 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" containerName="payload-processing" Oct 31 03:18:49.822310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822278 4854 cpu_manager.go:414] "containerMap: removing container" podUID="4db2452b-ccf0-41a2-a16a-a1fde0d1da65" containerName="payload-pre-processing" Oct 31 03:18:49.822310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822284 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db2452b-ccf0-41a2-a16a-a1fde0d1da65" containerName="payload-pre-processing" Oct 31 03:18:49.822536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822415 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="7776acb8-b5e8-44d7-8d07-e3b0788ab0f9" containerName="payload-processing" Oct 31 03:18:49.822536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822430 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db2452b-ccf0-41a2-a16a-a1fde0d1da65" containerName="payload-pre-processing" Oct 31 03:18:49.822536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.822439 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" containerName="istio-proxy" Oct 31 03:18:49.827713 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.827693 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.843842 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.843941 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.843974 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.844006 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.844062 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.844112 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.844143 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.844160 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.844215 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket\") pod \"728dec5f-bdf3-45ff-ab1c-133444da826c\" (UID: \"728dec5f-bdf3-45ff-ab1c-133444da826c\") " Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.845161 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.845494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.845449 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.846180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.845611 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.846180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.845743 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.851493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.847908 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:49.853976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.853954 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s"] Oct 31 03:18:49.854328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.854313 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3b8712-openshift-default-dockercfg-g67nt\"" Oct 31 03:18:49.855551 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.855479 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:49.875219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.875176 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:49.880659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.879303 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "kube-api-access-z4zg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:49.881889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.881865 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy" pod "728dec5f-bdf3-45ff-ab1c-133444da826c" (UID: "728dec5f-bdf3-45ff-ab1c-133444da826c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:49.893414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.891561 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.984446 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.984704 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:49.991299 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.003359 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021270 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021376 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mrz\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021411 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021514 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021536 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021732 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021745 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021759 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021773 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728dec5f-bdf3-45ff-ab1c-133444da826c-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021795 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021808 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zg7\" (UniqueName: \"kubernetes.io/projected/728dec5f-bdf3-45ff-ab1c-133444da826c-kube-api-access-z4zg7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021821 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021833 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.041282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.021846 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728dec5f-bdf3-45ff-ab1c-133444da826c-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.207073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.181458 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.256849 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.256816 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.257735 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.257713 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.258340 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.258323 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.258851 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.258825 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.259325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.258764 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:50.259325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.259186 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:50.259639 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.259615 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.260533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.260081 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:50.261112 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.261074 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:50.261268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.260493 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.261484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.261468 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.262082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.261960 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:50.262519 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.262032 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo\") pod \"5887b115-7ccf-41e8-b37f-eb5c2a326907\" (UID: \"5887b115-7ccf-41e8-b37f-eb5c2a326907\") " Oct 31 03:18:50.265038 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.265006 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.265199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.265180 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.265545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.265527 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.265999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.265888 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.266182 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.266143 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.267035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.267007 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.267332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.267305 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.267615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.267561 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mrz\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.267804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.267787 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.268071 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.268037 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.268219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.268201 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.268535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.268501 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.268719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.268705 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.268864 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.268831 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.268970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.268959 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5887b115-7ccf-41e8-b37f-eb5c2a326907-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.269093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.269064 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.271706 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.270948 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.272188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.272168 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.272692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.272673 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.279765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.279730 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "kube-api-access-6xlrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:50.279895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.279867 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:50.282297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.282224 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.347938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.342451 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:50.347938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.342616 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy" pod "5887b115-7ccf-41e8-b37f-eb5c2a326907" (UID: "5887b115-7ccf-41e8-b37f-eb5c2a326907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:50.392282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.385633 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.392282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.385668 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlrn\" (UniqueName: \"kubernetes.io/projected/5887b115-7ccf-41e8-b37f-eb5c2a326907-kube-api-access-6xlrn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.392282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.385684 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.392282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.385699 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5887b115-7ccf-41e8-b37f-eb5c2a326907-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:50.439340 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.439304 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs"] Oct 31 03:18:50.450952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.450609 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.456939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.456888 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mrz\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.459753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.459691 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token\") pod \"e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.466196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.466174 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-4rxjs"] Oct 31 03:18:50.488445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.488412 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:50.500830 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.500797 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs"] Oct 31 03:18:50.502386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.502361 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" Oct 31 03:18:50.502497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.502387 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" Oct 31 03:18:50.502819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.502798 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" containerName="istio-proxy" Oct 31 03:18:50.509428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.509403 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.512394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.512369 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9"] Oct 31 03:18:50.518483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.518467 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b"] Oct 31 03:18:50.518628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.518596 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.526237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.526214 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs"] Oct 31 03:18:50.526424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.526361 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.535219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.535149 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b"] Oct 31 03:18:50.542819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.542795 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9"] Oct 31 03:18:50.588778 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588670 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.588778 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588713 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.588778 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588738 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.588778 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588758 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588802 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588832 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588858 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588890 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588934 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.588967 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589000 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589023 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589045 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589080 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589095 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589110 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589142 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2db6x\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589167 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589189 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589212 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589238 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589282 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnl4b\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589299 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589355 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589395 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589429 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.589555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.589449 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzck\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.712459 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.712369 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.713188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.713162 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.713413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.713391 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.713716 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.713639 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.713891 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.713873 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.714046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.714031 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.714146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.714134 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.714646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.714494 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.714905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.714875 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2db6x\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.715046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715031 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.715215 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715200 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.715360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715344 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.715577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715561 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.715677 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715664 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnl4b\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.715780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715768 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.715950 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.715935 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.716095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716081 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.716207 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716194 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.716344 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716330 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzck\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.716510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716498 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.716639 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716623 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.716754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716740 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.716852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716840 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.717010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.716997 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.717124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.717112 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.717228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.717216 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.717373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.717359 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.720752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.719636 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.720752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.713085 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.720752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.720020 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.726625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.723357 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.726625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.723621 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.726625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.725559 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.726862 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.726647 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.728938 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.728082 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.732530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.732498 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.739382 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.739045 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.739382 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.739336 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.739612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.739561 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.739803 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.739782 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.740449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.740424 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.752082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.751675 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.765534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.756700 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2db6x\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.765534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.759695 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.765534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.760059 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.765534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.761220 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.766607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.766517 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.769179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.767973 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.769179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.768064 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzck\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.769781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.769383 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnl4b\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.769781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.769578 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.769781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.769742 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:50.769781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.769489 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.771947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.771863 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.802671 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.802647 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" Oct 31 03:18:50.802671 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.802654 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b" event={"ID":"5887b115-7ccf-41e8-b37f-eb5c2a326907","Type":"ContainerDied","Data":"1a39f0cfa1b02f034e5b923614ac58427900b340af3c34b79b513fec1d980f92"} Oct 31 03:18:50.802861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.802726 4854 scope.go:122] "RemoveContainer" containerID="499bf728150767fb640e00565fa151966a74a3f7b73fe8fd806fbc3a318f7f71" Oct 31 03:18:50.822233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.822210 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s"] Oct 31 03:18:50.824281 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:50.824238 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0b9870_c9ba_4005_8a97_417ca943412b.slice/crio-5ca0490e31505d635953d13a37c1735c40c8ac7c7feda9c071a6ea92e5bc36ab WatchSource:0}: Error finding container 5ca0490e31505d635953d13a37c1735c40c8ac7c7feda9c071a6ea92e5bc36ab: Status 404 returned error can't find the container with id 5ca0490e31505d635953d13a37c1735c40c8ac7c7feda9c071a6ea92e5bc36ab Oct 31 03:18:50.826905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.826887 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b"] Oct 31 03:18:50.828036 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.828004 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:50.828107 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.828091 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:50.828153 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.828136 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:50.831065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.831049 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-gp22b"] Oct 31 03:18:50.869630 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.869500 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:50.881675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.881635 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:50.888407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:50.888381 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:51.201777 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:51.201734 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636c573f_b619_4775_984c_1f0ed67fb680.slice/crio-12b7cacadd20a3dd5dcf2c1ca1b28f71f871d22b5870f97acaa91fd46ce80b1c WatchSource:0}: Error finding container 12b7cacadd20a3dd5dcf2c1ca1b28f71f871d22b5870f97acaa91fd46ce80b1c: Status 404 returned error can't find the container with id 12b7cacadd20a3dd5dcf2c1ca1b28f71f871d22b5870f97acaa91fd46ce80b1c Oct 31 03:18:51.204155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.204119 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs"] Oct 31 03:18:51.205229 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:51.205204 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54e50f9_15a5_4a67_a11d_6d9c806699cc.slice/crio-220eabbc01155ba624e024127303ddef7d618599d1fabe08cc17eccda7774264 WatchSource:0}: Error finding container 220eabbc01155ba624e024127303ddef7d618599d1fabe08cc17eccda7774264: Status 404 returned error can't find the container with id 220eabbc01155ba624e024127303ddef7d618599d1fabe08cc17eccda7774264 Oct 31 03:18:51.205397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.205373 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.205448 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.205434 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.205483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.205469 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.207975 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.207942 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.208104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.208020 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.208104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.208065 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.209766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.209745 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9"] Oct 31 03:18:51.220670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.220624 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b"] Oct 31 03:18:51.228507 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:18:51.228478 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4aec8c4_40ac_4696_bdd5_ed3ec625e576.slice/crio-8859eed02ba035c3a2429a5191c0d7d30e79e7fb9b7c98943653316d136dd0c6 WatchSource:0}: Error finding container 8859eed02ba035c3a2429a5191c0d7d30e79e7fb9b7c98943653316d136dd0c6: Status 404 returned error can't find the container with id 8859eed02ba035c3a2429a5191c0d7d30e79e7fb9b7c98943653316d136dd0c6 Oct 31 03:18:51.232589 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.231944 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.232675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.232641 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:51.232750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:51.232686 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:18:52.045275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.038148 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" event={"ID":"636c573f-b619-4775-984c-1f0ed67fb680","Type":"ContainerStarted","Data":"d92939840b049badf6ec681406326ab96cb806bc313131f763c55dc80f79b4fc"} Oct 31 03:18:52.045275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.038281 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" event={"ID":"636c573f-b619-4775-984c-1f0ed67fb680","Type":"ContainerStarted","Data":"12b7cacadd20a3dd5dcf2c1ca1b28f71f871d22b5870f97acaa91fd46ce80b1c"} Oct 31 03:18:52.051306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.051126 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" event={"ID":"3b0b9870-c9ba-4005-8a97-417ca943412b","Type":"ContainerStarted","Data":"47d5d92bb5954d9f0a9926d125ca34b8911ee44a3be0c2e36df3af31af7df39b"} Oct 31 03:18:52.051306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.051191 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" event={"ID":"3b0b9870-c9ba-4005-8a97-417ca943412b","Type":"ContainerStarted","Data":"5ca0490e31505d635953d13a37c1735c40c8ac7c7feda9c071a6ea92e5bc36ab"} Oct 31 03:18:52.055927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.055899 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" event={"ID":"f54e50f9-15a5-4a67-a11d-6d9c806699cc","Type":"ContainerStarted","Data":"75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c"} Oct 31 03:18:52.056096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.056082 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" event={"ID":"f54e50f9-15a5-4a67-a11d-6d9c806699cc","Type":"ContainerStarted","Data":"220eabbc01155ba624e024127303ddef7d618599d1fabe08cc17eccda7774264"} Oct 31 03:18:52.059024 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.058990 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" event={"ID":"b4aec8c4-40ac-4696-bdd5-ed3ec625e576","Type":"ContainerStarted","Data":"9b8b47bcb6dbf8d45cc6c0d48fa58bf9f2316e75ea927291160685c3d9895ed1"} Oct 31 03:18:52.059120 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.059031 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" event={"ID":"b4aec8c4-40ac-4696-bdd5-ed3ec625e576","Type":"ContainerStarted","Data":"8859eed02ba035c3a2429a5191c0d7d30e79e7fb9b7c98943653316d136dd0c6"} Oct 31 03:18:52.066411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.065004 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podStartSLOduration=2.064988106 podStartE2EDuration="2.064988106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:52.060810959 +0000 UTC m=+1415.757978268" watchObservedRunningTime="2026-10-31 03:18:52.064988106 +0000 UTC m=+1415.762155414" Oct 31 03:18:52.086407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.086366 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podStartSLOduration=3.086354016 podStartE2EDuration="3.086354016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:52.081110875 +0000 UTC m=+1415.778278180" watchObservedRunningTime="2026-10-31 03:18:52.086354016 +0000 UTC m=+1415.783521378" Oct 31 03:18:52.121434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.121384 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podStartSLOduration=2.121371566 podStartE2EDuration="2.121371566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:52.117148234 +0000 UTC m=+1415.814315535" watchObservedRunningTime="2026-10-31 03:18:52.121371566 +0000 UTC m=+1415.818538869" Oct 31 03:18:52.141131 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.141084 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podStartSLOduration=2.141067678 podStartE2EDuration="2.141067678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:52.13978225 +0000 UTC m=+1415.836949551" watchObservedRunningTime="2026-10-31 03:18:52.141067678 +0000 UTC m=+1415.838234983" Oct 31 03:18:52.459126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.459039 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5887b115-7ccf-41e8-b37f-eb5c2a326907" path="/var/lib/kubelet/pods/5887b115-7ccf-41e8-b37f-eb5c2a326907/volumes" Oct 31 03:18:52.459726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.459707 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728dec5f-bdf3-45ff-ab1c-133444da826c" path="/var/lib/kubelet/pods/728dec5f-bdf3-45ff-ab1c-133444da826c/volumes" Oct 31 03:18:52.489163 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.489137 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:52.494755 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.494732 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:52.870381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.870350 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:18:52.871666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.871646 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:52.871715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.871693 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:52.882363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.882332 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:18:52.883395 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.883370 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:52.883522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.883414 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:52.888888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.888866 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:18:52.890110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.890081 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:52.890226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:52.890136 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:53.130334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.129869 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:53.135003 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.134614 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:18:53.876270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.872684 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:53.876270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.872758 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:53.906293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.891183 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:53.906293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.891322 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:53.906293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.891658 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:53.906293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:53.891689 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:54.355406 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.348341 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 03:18:54.367445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.367401 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:54.871572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.871524 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:54.881132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.872268 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:54.881401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.881365 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:54.884333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.884312 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:54.884537 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.884514 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:54.888854 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.888832 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:54.889030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:54.889007 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:55.887405 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.887369 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:55.911301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.889189 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:55.911301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.888627 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:55.911301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.889491 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:55.911301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.888665 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:55.911301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.889540 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:56.178449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:55.951760 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:18:56.770297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.767900 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r"] Oct 31 03:18:56.779717 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.779690 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r"] Oct 31 03:18:56.779922 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.779823 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:56.831522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.831490 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3b8712\"" Oct 31 03:18:56.872195 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.872161 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:56.872490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.872458 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:56.883484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.883450 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:56.883599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.883526 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:56.884105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.884083 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:56.884338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.884324 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp4k\" (UniqueName: \"kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k\") pod \"payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:56.894012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.893989 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:56.912341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:56.912303 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:57.003283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.001076 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:57.003283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.001232 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp4k\" (UniqueName: \"kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k\") pod \"payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:57.004470 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.004442 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:57.144317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.144163 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp4k\" (UniqueName: \"kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k\") pod \"payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:57.444324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.443478 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:57.888523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.885760 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:57.888523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.885824 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:57.888523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.886881 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:57.888523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:57.886932 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:58.044554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.044372 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:58.044554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.044437 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:58.485916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.485892 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r"] Oct 31 03:18:58.884189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.884070 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:58.884189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.884125 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:58.894194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.893919 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:58.894194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.893970 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:18:58.985275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.944087 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:58.985275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:58.944162 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:59.228337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.227265 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" event={"ID":"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035","Type":"ContainerStarted","Data":"df7c81ca5e00b52de795527a2d72afd2b7b932f9a87f12c3cdd4e23deebcc09a"} Oct 31 03:18:59.228337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.227421 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" event={"ID":"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035","Type":"ContainerStarted","Data":"5c4d107d139ecd24e0cac504f508b1d382dc02916437e69921e79f7dbcdccc64"} Oct 31 03:18:59.240276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.229047 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:18:59.361276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.357041 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" podStartSLOduration=4.357018312 podStartE2EDuration="4.357018312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.314731512 +0000 UTC m=+1423.011898817" watchObservedRunningTime="2026-10-31 03:18:59.357018312 +0000 UTC m=+1423.054185618" Oct 31 03:18:59.899282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.899106 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:18:59.899282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.899178 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:18:59.902280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.900953 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:18:59.902280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.901004 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:18:59.902280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.900897 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:18:59.902280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:18:59.901081 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:19:00.336287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.333849 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:00.341284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.339352 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:00.899358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.899304 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:19:00.919278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.908642 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:19:00.919555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.919523 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 03:19:00.923276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.910339 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 03:19:00.923276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.919693 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 03:19:00.939397 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.939366 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:19:00.939643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:00.939620 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 03:19:01.046750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.046704 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:01.192332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.189150 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:01.252004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.250999 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:01.312275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.308529 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:01.350273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.340141 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:19:01.350273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.347162 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:19:01.519274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.510122 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:19:02.037650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.987646 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:19:02.037650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:01.989447 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:19:02.037650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.002567 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:19:02.261544 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.261515 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:19:02.262313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.262291 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:19:02.263549 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.263530 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:19:02.278101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.278078 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:19:02.278523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.278505 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:19:02.279314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:02.279287 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:19:09.257166 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:19:09.257128 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:19:11.270158 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.270122 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b"] Oct 31 03:19:11.276855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.276829 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.279981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.279955 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7269ae-openshift-default-dockercfg-kfww6\"" Oct 31 03:19:11.290010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.289987 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b"] Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.402916 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.402987 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403030 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403079 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403200 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5f6\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403401 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403440 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403500 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.407280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.403591 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.461961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.461923 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514357 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514588 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514634 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514672 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514716 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514813 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5f6\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514859 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514883 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.515275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.514929 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.571323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.563754 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.572307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.569881 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.572488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.570049 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.572592 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.570627 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.600277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.586722 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.605347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.605305 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.637784 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.637733 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:11.819302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:11.804074 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5f6\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:12.130767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:12.066094 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:12.218704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:12.217706 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:12.857495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:12.857387 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b"] Oct 31 03:19:12.863208 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:19:12.863170 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ebd920_0c0a_4cee_b2ef_8b307ce014f3.slice/crio-31e51f33528dda0a35ba2acc377f423a5748fd695ba8e4a0433cced2a3c4d794 WatchSource:0}: Error finding container 31e51f33528dda0a35ba2acc377f423a5748fd695ba8e4a0433cced2a3c4d794: Status 404 returned error can't find the container with id 31e51f33528dda0a35ba2acc377f423a5748fd695ba8e4a0433cced2a3c4d794 Oct 31 03:19:12.866285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:12.866234 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:12.866385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:12.866327 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:12.866385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:12.866364 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:13.083433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:13.083395 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" event={"ID":"78ebd920-0c0a-4cee-b2ef-8b307ce014f3","Type":"ContainerStarted","Data":"7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674"} Oct 31 03:19:13.083625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:13.083457 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" event={"ID":"78ebd920-0c0a-4cee-b2ef-8b307ce014f3","Type":"ContainerStarted","Data":"31e51f33528dda0a35ba2acc377f423a5748fd695ba8e4a0433cced2a3c4d794"} Oct 31 03:19:13.106097 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:13.106041 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podStartSLOduration=2.106028594 podStartE2EDuration="2.106028594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:13.10388126 +0000 UTC m=+1436.801048565" watchObservedRunningTime="2026-10-31 03:19:13.106028594 +0000 UTC m=+1436.803195898" Oct 31 03:19:13.236328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:13.229641 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:13.236328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:13.231438 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:19:13.236328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:13.231496 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:19:14.222879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:14.222851 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:15.100189 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:15.100157 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:15.101012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:15.100994 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:19:16.671556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.671521 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9"] Oct 31 03:19:16.678149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.678126 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9"] Oct 31 03:19:16.678334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.678233 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.713954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.713931 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6d1a8e-openshift-default-dockercfg-bjjnf\"" Oct 31 03:19:16.744358 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744332 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744459 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744404 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744542 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744604 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744574 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744656 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744643 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46v69\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744678 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744763 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744848 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.744919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.744907 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.846628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846596 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.846628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846634 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.846872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846693 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46v69\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.846872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846737 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.846872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846796 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847043 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846893 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847043 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.846968 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847071 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847102 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847321 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847299 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847377 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847544 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847497 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847507 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.847838 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.847817 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.849720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.849696 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.849852 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.849836 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.885172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.885144 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46v69\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.886570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.886549 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:16.992864 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:16.992836 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:17.291109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:17.291070 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9"] Oct 31 03:19:17.294499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:17.294467 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:17.294607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:17.294546 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:17.294607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:17.294594 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:18.161310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:18.161271 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" event={"ID":"02495fbc-ba01-4465-a2d9-d64b540c47ee","Type":"ContainerStarted","Data":"8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d"} Oct 31 03:19:18.161799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:18.161372 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" event={"ID":"02495fbc-ba01-4465-a2d9-d64b540c47ee","Type":"ContainerStarted","Data":"bf31a4f048bcca161d3ca420079cc22dd4eb4faa019d04026de26f2a418fad50"} Oct 31 03:19:18.237273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:18.232901 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" podStartSLOduration=2.232880802 podStartE2EDuration="2.232880802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:18.215619204 +0000 UTC m=+1441.912786509" watchObservedRunningTime="2026-10-31 03:19:18.232880802 +0000 UTC m=+1441.930048113" Oct 31 03:19:18.519761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:18.519727 4854 scope.go:122] "RemoveContainer" containerID="3020816f91b613261defcb93217e21653baee2c309c53d951445945ec951937e" Oct 31 03:19:19.001264 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:18.999709 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:19.005565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:19.002784 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:19:19.005565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:19.002840 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 03:19:19.997272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:19.996611 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:19:19.997272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:19.996673 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 03:19:20.260623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:20.259665 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:20.997840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:20.997110 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:19:20.997840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:20.997174 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 03:19:22.019276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:22.015736 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:19:22.019276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:22.015794 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 03:19:22.472587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:22.472504 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9"] Oct 31 03:19:22.472801 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:22.472772 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" containerID="cri-o://8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d" gracePeriod=30 Oct 31 03:19:27.964792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:27.964757 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw"] Oct 31 03:19:27.986835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:27.986622 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn"] Oct 31 03:19:29.416089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.411380 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs"] Oct 31 03:19:29.416089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.412061 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" containerID="cri-o://d92939840b049badf6ec681406326ab96cb806bc313131f763c55dc80f79b4fc" gracePeriod=30 Oct 31 03:19:29.454580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.453544 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s"] Oct 31 03:19:29.454580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.453629 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9"] Oct 31 03:19:29.454580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.453865 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" containerID="cri-o://75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c" gracePeriod=30 Oct 31 03:19:29.454580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.454170 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" containerID="cri-o://52a79d381256f76392e11da477f674238d53729bf830a6aaf4042d4321ddfc03" gracePeriod=30 Oct 31 03:19:29.465560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.459006 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b"] Oct 31 03:19:29.465560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:29.459207 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" containerID="cri-o://9b8b47bcb6dbf8d45cc6c0d48fa58bf9f2316e75ea927291160685c3d9895ed1" gracePeriod=30 Oct 31 03:19:30.449134 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.449104 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:30.593290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.591516 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.593290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.592162 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.595652 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.595623 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:30.598878 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.598848 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.599034 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599010 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.599386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599364 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.599722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599572 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.599722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599591 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.599722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599643 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.599913 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599770 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.599913 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.599848 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.600336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.600011 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69\") pod \"02495fbc-ba01-4465-a2d9-d64b540c47ee\" (UID: \"02495fbc-ba01-4465-a2d9-d64b540c47ee\") " Oct 31 03:19:30.600336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.600059 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.601605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.601092 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:30.601605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.601333 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.609096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.603051 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.611066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.611024 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.611262 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.611230 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.611430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.611414 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.611753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.611736 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02495fbc-ba01-4465-a2d9-d64b540c47ee-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.611952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.611936 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.614707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.614626 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:30.618396 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.618352 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.619060 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.618979 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69" pod "02495fbc-ba01-4465-a2d9-d64b540c47ee" (UID: "02495fbc-ba01-4465-a2d9-d64b540c47ee"). InnerVolumeSpecName "kube-api-access-46v69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:30.714229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.714153 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.714229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.714186 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46v69\" (UniqueName: \"kubernetes.io/projected/02495fbc-ba01-4465-a2d9-d64b540c47ee-kube-api-access-46v69\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.714229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.714198 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02495fbc-ba01-4465-a2d9-d64b540c47ee-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.854008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.853973 4854 generic.go:356] "Generic (PLEG): container finished" podID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerID="8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d" exitCode=0 Oct 31 03:19:30.856240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.854036 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" Oct 31 03:19:30.856240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.854040 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" event={"ID":"02495fbc-ba01-4465-a2d9-d64b540c47ee","Type":"ContainerDied","Data":"8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d"} Oct 31 03:19:30.856240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.854154 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9" event={"ID":"02495fbc-ba01-4465-a2d9-d64b540c47ee","Type":"ContainerDied","Data":"bf31a4f048bcca161d3ca420079cc22dd4eb4faa019d04026de26f2a418fad50"} Oct 31 03:19:30.856240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.854174 4854 scope.go:122] "RemoveContainer" containerID="8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d" Oct 31 03:19:30.867288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.867230 4854 scope.go:122] "RemoveContainer" containerID="8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d" Oct 31 03:19:30.867601 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:19:30.867577 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d\": container with ID starting with 8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d not found: ID does not exist" containerID="8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d" Oct 31 03:19:30.867680 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.867612 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d"} err="failed to get container status \"8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d\": rpc error: code = NotFound desc = could not find container \"8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d\": container with ID starting with 8c739eb92b7e1e21a1eae1ec8f456a93d2a8935c1afbee46e49ac321e411f85d not found: ID does not exist" Oct 31 03:19:30.880202 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.880175 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9"] Oct 31 03:19:30.886127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:30.886102 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-jf2p9"] Oct 31 03:19:31.881597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.881558 4854 generic.go:356] "Generic (PLEG): container finished" podID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerID="714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e" exitCode=0 Oct 31 03:19:31.882114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.881676 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" event={"ID":"9661d7af-7aba-4d48-bf9a-1f97f804b29a","Type":"ContainerDied","Data":"714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e"} Oct 31 03:19:31.882198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.882176 4854 scope.go:122] "RemoveContainer" containerID="714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e" Oct 31 03:19:31.884436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.884405 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:31.884538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.884476 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:31.884575 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.884509 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:31.992009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.991981 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:19:31.992148 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:31.992037 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:19:32.264302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.264219 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:19:32.264496 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.264313 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:19:32.281041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.281016 4854 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:19:32.281183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.281054 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:19:32.408632 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.408597 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg"] Oct 31 03:19:32.409933 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.409664 4854 cpu_manager.go:414] "containerMap: removing container" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" Oct 31 03:19:32.409933 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.409695 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" Oct 31 03:19:32.410143 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.410004 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" containerName="istio-proxy" Oct 31 03:19:32.414666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.414641 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.417132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.417107 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg"] Oct 31 03:19:32.420679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.420658 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fccff5-openshift-default-dockercfg-bhk57\"" Oct 31 03:19:32.459822 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.459796 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02495fbc-ba01-4465-a2d9-d64b540c47ee" path="/var/lib/kubelet/pods/02495fbc-ba01-4465-a2d9-d64b540c47ee/volumes" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.604656 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.604746 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.604790 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.604833 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.605014 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.605045 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.605092 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.605155 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9ms\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.606322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.605276 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.715072 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.715127 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.715183 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.716079 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.716927 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.717023 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9ms\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.717377 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.717550 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.717604 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.717646 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.720292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.717689 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.747270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.737317 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.747270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.737772 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.747270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.738375 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.764289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.760774 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.767894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.767792 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.777273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.775860 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:32.811356 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:32.803744 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9ms\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms\") pod \"e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:33.018813 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.018628 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" event={"ID":"9661d7af-7aba-4d48-bf9a-1f97f804b29a","Type":"ContainerStarted","Data":"29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d"} Oct 31 03:19:33.044906 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.044867 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:33.470860 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:19:33.470795 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45e38e9_1553_4655_ae81_0875d884214e.slice/crio-22d133c27e7010c35eeb0b59e5830e3fd5355784759a60a48077666635ca30f6 WatchSource:0}: Error finding container 22d133c27e7010c35eeb0b59e5830e3fd5355784759a60a48077666635ca30f6: Status 404 returned error can't find the container with id 22d133c27e7010c35eeb0b59e5830e3fd5355784759a60a48077666635ca30f6 Oct 31 03:19:33.472468 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.472018 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg"] Oct 31 03:19:33.475511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.475267 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:33.475511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.475356 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:33.475511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.475402 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:33.479191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.479167 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:19:33.482417 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.482390 4854 patch_prober.go:29] interesting pod/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:19:33.482502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.482451 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 03:19:33.540438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.540306 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9"] Oct 31 03:19:33.544961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:33.544938 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt"] Oct 31 03:19:34.104277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.096605 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" event={"ID":"a45e38e9-1553-4655-ae81-0875d884214e","Type":"ContainerStarted","Data":"6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c"} Oct 31 03:19:34.186026 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.185999 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:34.186333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.186309 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" event={"ID":"a45e38e9-1553-4655-ae81-0875d884214e","Type":"ContainerStarted","Data":"22d133c27e7010c35eeb0b59e5830e3fd5355784759a60a48077666635ca30f6"} Oct 31 03:19:34.186485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.165610 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podStartSLOduration=2.165583222 podStartE2EDuration="2.165583222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:34.144633795 +0000 UTC m=+1457.841801104" watchObservedRunningTime="2026-10-31 03:19:34.165583222 +0000 UTC m=+1457.862750526" Oct 31 03:19:34.187700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.185911 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:19:34.188171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.188148 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:19:34.498593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.490418 4854 patch_prober.go:29] interesting pod/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:19:34.498593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:34.490477 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 03:19:35.056933 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.056599 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:35.075030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.071655 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088160 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088224 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088623 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088709 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088789 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088817 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088862 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088925 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.088962 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.101434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.089179 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo\") pod \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\" (UID: \"f54e50f9-15a5-4a67-a11d-6d9c806699cc\") " Oct 31 03:19:35.134313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.117166 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.134313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.132672 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.148322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.135621 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.148322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.136539 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.148322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.140068 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.151513 4854 generic.go:356] "Generic (PLEG): container finished" podID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerID="75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c" exitCode=0 Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.151665 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" event={"ID":"f54e50f9-15a5-4a67-a11d-6d9c806699cc","Type":"ContainerDied","Data":"75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c"} Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.151718 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" event={"ID":"f54e50f9-15a5-4a67-a11d-6d9c806699cc","Type":"ContainerDied","Data":"220eabbc01155ba624e024127303ddef7d618599d1fabe08cc17eccda7774264"} Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.151749 4854 scope.go:122] "RemoveContainer" containerID="75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c" Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.151883 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9" Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.156620 4854 generic.go:356] "Generic (PLEG): container finished" podID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerID="9b8b47bcb6dbf8d45cc6c0d48fa58bf9f2316e75ea927291160685c3d9895ed1" exitCode=0 Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.156782 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" event={"ID":"b4aec8c4-40ac-4696-bdd5-ed3ec625e576","Type":"ContainerDied","Data":"9b8b47bcb6dbf8d45cc6c0d48fa58bf9f2316e75ea927291160685c3d9895ed1"} Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.158959 4854 generic.go:356] "Generic (PLEG): container finished" podID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerID="52a79d381256f76392e11da477f674238d53729bf830a6aaf4042d4321ddfc03" exitCode=0 Oct 31 03:19:35.159302 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.159110 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" event={"ID":"0bc9ef1f-7391-48a9-af37-186eaa30f981","Type":"ContainerDied","Data":"52a79d381256f76392e11da477f674238d53729bf830a6aaf4042d4321ddfc03"} Oct 31 03:19:35.162027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.161150 4854 generic.go:356] "Generic (PLEG): container finished" podID="636c573f-b619-4775-984c-1f0ed67fb680" containerID="d92939840b049badf6ec681406326ab96cb806bc313131f763c55dc80f79b4fc" exitCode=0 Oct 31 03:19:35.162027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.161956 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" event={"ID":"636c573f-b619-4775-984c-1f0ed67fb680","Type":"ContainerDied","Data":"d92939840b049badf6ec681406326ab96cb806bc313131f763c55dc80f79b4fc"} Oct 31 03:19:35.162353 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.162304 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:35.163923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.163673 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:19:35.164732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.164705 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "kube-api-access-2db6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.166396 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.166367 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.166866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.166833 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.169049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.169028 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo" pod "f54e50f9-15a5-4a67-a11d-6d9c806699cc" (UID: "f54e50f9-15a5-4a67-a11d-6d9c806699cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229224 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229277 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229293 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2db6x\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-kube-api-access-2db6x\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229307 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229321 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229334 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229347 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.231596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.229361 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54e50f9-15a5-4a67-a11d-6d9c806699cc-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.254626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.254421 4854 scope.go:122] "RemoveContainer" containerID="75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c" Oct 31 03:19:35.255015 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:19:35.254996 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c\": container with ID starting with 75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c not found: ID does not exist" containerID="75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c" Oct 31 03:19:35.255166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.255145 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c"} err="failed to get container status \"75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c\": rpc error: code = NotFound desc = could not find container \"75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c\": container with ID starting with 75ba65e05e4395d8c35f8f507da1c95c0074f7727a29bd09bcd8698d5bbc2a4c not found: ID does not exist" Oct 31 03:19:35.262555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.262533 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:19:35.271285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.270066 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:19:35.272323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.272235 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:19:35.335421 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335321 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335421 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335376 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.335655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335428 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.335655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335456 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.335655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335497 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.335655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335537 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335593 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335618 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335672 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335760 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335783 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335826 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335906 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335927 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335952 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.335992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.335992 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336010 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336073 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336096 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336159 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336179 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336278 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data\") pod \"636c573f-b619-4775-984c-1f0ed67fb680\" (UID: \"636c573f-b619-4775-984c-1f0ed67fb680\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336350 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336395 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336431 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data\") pod \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\" (UID: \"b4aec8c4-40ac-4696-bdd5-ed3ec625e576\") " Oct 31 03:19:35.336481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336453 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.336948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.336513 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo\") pod \"0bc9ef1f-7391-48a9-af37-186eaa30f981\" (UID: \"0bc9ef1f-7391-48a9-af37-186eaa30f981\") " Oct 31 03:19:35.337615 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.337590 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.337744 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.337609 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.338046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.338028 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.338285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.338242 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.338494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.338472 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.338600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.338479 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.338670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.338632 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.338954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.338935 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.339381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.339359 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.339505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.339479 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.340056 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.339675 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.340203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.339763 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.340316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.339859 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.348477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.348447 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.354374 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.348446 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.354550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.339488 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.358034 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.354672 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.365186 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.365214 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.365233 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.365270 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.365285 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.363558 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.373412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.368087 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "kube-api-access-dnl4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.414275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.413462 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.441578 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.441518 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.445083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.445017 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.445484 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.445434 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "kube-api-access-65bvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.445754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.445716 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.446116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.446079 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "kube-api-access-9pzck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.446455 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.446411 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.446681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.446650 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.446897 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.446869 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy" pod "b4aec8c4-40ac-4696-bdd5-ed3ec625e576" (UID: "b4aec8c4-40ac-4696-bdd5-ed3ec625e576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.447119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.447091 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token" pod "636c573f-b619-4775-984c-1f0ed67fb680" (UID: "636c573f-b619-4775-984c-1f0ed67fb680"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.487335 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.487401 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.487434 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636c573f-b619-4775-984c-1f0ed67fb680-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.487453 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.487467 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492369 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492434 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492452 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492467 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492481 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492498 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492515 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc9ef1f-7391-48a9-af37-186eaa30f981-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492531 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636c573f-b619-4775-984c-1f0ed67fb680-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492543 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnl4b\" (UniqueName: \"kubernetes.io/projected/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-kube-api-access-dnl4b\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492558 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bvq\" (UniqueName: \"kubernetes.io/projected/0bc9ef1f-7391-48a9-af37-186eaa30f981-kube-api-access-65bvq\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492574 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4aec8c4-40ac-4696-bdd5-ed3ec625e576-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492604 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636c573f-b619-4775-984c-1f0ed67fb680-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.492621 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzck\" (UniqueName: \"kubernetes.io/projected/636c573f-b619-4775-984c-1f0ed67fb680-kube-api-access-9pzck\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.495847 4854 patch_prober.go:29] interesting pod/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.495898 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 03:19:35.513273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.496383 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo" pod "0bc9ef1f-7391-48a9-af37-186eaa30f981" (UID: "0bc9ef1f-7391-48a9-af37-186eaa30f981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.541115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.541072 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9"] Oct 31 03:19:35.547443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.547412 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-5gjn9"] Oct 31 03:19:35.614490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:35.606106 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc9ef1f-7391-48a9-af37-186eaa30f981-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:36.169804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.169764 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" Oct 31 03:19:36.170342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.169801 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs" event={"ID":"636c573f-b619-4775-984c-1f0ed67fb680","Type":"ContainerDied","Data":"12b7cacadd20a3dd5dcf2c1ca1b28f71f871d22b5870f97acaa91fd46ce80b1c"} Oct 31 03:19:36.170342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.169896 4854 scope.go:122] "RemoveContainer" containerID="d92939840b049badf6ec681406326ab96cb806bc313131f763c55dc80f79b4fc" Oct 31 03:19:36.172921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.172903 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" Oct 31 03:19:36.173048 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.172893 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b" event={"ID":"b4aec8c4-40ac-4696-bdd5-ed3ec625e576","Type":"ContainerDied","Data":"8859eed02ba035c3a2429a5191c0d7d30e79e7fb9b7c98943653316d136dd0c6"} Oct 31 03:19:36.175376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.175354 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" Oct 31 03:19:36.177218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.177193 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s" event={"ID":"0bc9ef1f-7391-48a9-af37-186eaa30f981","Type":"ContainerDied","Data":"e44ba543095eb6b51ad1c48b82377095bcfaf2013df6a7a45847a232abbbdb75"} Oct 31 03:19:36.184230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.184211 4854 scope.go:122] "RemoveContainer" containerID="9b8b47bcb6dbf8d45cc6c0d48fa58bf9f2316e75ea927291160685c3d9895ed1" Oct 31 03:19:36.226616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.226584 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs"] Oct 31 03:19:36.230163 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.230136 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-mckbs"] Oct 31 03:19:36.232233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.232214 4854 scope.go:122] "RemoveContainer" containerID="52a79d381256f76392e11da477f674238d53729bf830a6aaf4042d4321ddfc03" Oct 31 03:19:36.242668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.242648 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b"] Oct 31 03:19:36.246553 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.246531 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-pfp9b"] Oct 31 03:19:36.255410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.255390 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s"] Oct 31 03:19:36.261754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.261732 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-8r67s"] Oct 31 03:19:36.478613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.478545 4854 patch_prober.go:29] interesting pod/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:19:36.478613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.478597 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 03:19:36.481970 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.481942 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" path="/var/lib/kubelet/pods/0bc9ef1f-7391-48a9-af37-186eaa30f981/volumes" Oct 31 03:19:36.482727 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.482708 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636c573f-b619-4775-984c-1f0ed67fb680" path="/var/lib/kubelet/pods/636c573f-b619-4775-984c-1f0ed67fb680/volumes" Oct 31 03:19:36.483406 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.483388 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" path="/var/lib/kubelet/pods/b4aec8c4-40ac-4696-bdd5-ed3ec625e576/volumes" Oct 31 03:19:36.484522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:36.484503 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" path="/var/lib/kubelet/pods/f54e50f9-15a5-4a67-a11d-6d9c806699cc/volumes" Oct 31 03:19:37.478123 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:37.478088 4854 patch_prober.go:29] interesting pod/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:19:37.478539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:37.478141 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 03:19:38.501277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:38.486077 4854 patch_prober.go:29] interesting pod/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:19:38.501277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:38.486148 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 03:19:38.567276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:38.543401 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerName="payload-processing" containerID="cri-o://e729d68c2cc60bdd87b1c5d3b6e0f4658803a8757c091ee3aad1972bfc11bf7d" gracePeriod=25 Oct 31 03:19:38.567276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:38.546084 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerName="payload-pre-processing" containerID="cri-o://aad765d2c46678ea3f3b42ad4da24246d0d66383d2e0669de43fbac2f591adcc" gracePeriod=25 Oct 31 03:19:39.253675 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:19:39.253620 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:19:39.490273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:39.489427 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:19:39.528367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:39.526640 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:19:40.423115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:40.423085 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:19:41.371273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:41.370203 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:19:41.371273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:41.370608 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:19:42.317878 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:42.317837 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs"] Oct 31 03:19:42.323284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:42.323241 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk"] Oct 31 03:19:42.987886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:42.987831 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" containerID="cri-o://575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03" gracePeriod=45 Oct 31 03:19:43.069276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.063889 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" containerID="cri-o://a8079303242ca717008cc7a02c2fa23211a28027ea77f6dd36eaae08bd31f474" gracePeriod=45 Oct 31 03:19:43.237030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.236986 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused" Oct 31 03:19:43.384027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.383994 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:19:43.463458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463432 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache\") pod \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " Oct 31 03:19:43.463655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463533 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb\" (UniqueName: \"kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb\") pod \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " Oct 31 03:19:43.463655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463606 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm\") pod \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " Oct 31 03:19:43.463655 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463640 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache" pod "82cafdf5-57b6-46bd-9dd0-260613f4ccf2" (UID: "82cafdf5-57b6-46bd-9dd0-260613f4ccf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.463834 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463668 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir\") pod \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " Oct 31 03:19:43.463834 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463714 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home\") pod \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " Oct 31 03:19:43.463994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.463977 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs\" (UniqueName: \"kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs\") pod \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\" (UID: \"82cafdf5-57b6-46bd-9dd0-260613f4ccf2\") " Oct 31 03:19:43.464079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.464055 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir" pod "82cafdf5-57b6-46bd-9dd0-260613f4ccf2" (UID: "82cafdf5-57b6-46bd-9dd0-260613f4ccf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.464235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.464220 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home" pod "82cafdf5-57b6-46bd-9dd0-260613f4ccf2" (UID: "82cafdf5-57b6-46bd-9dd0-260613f4ccf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.465167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.464932 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.465167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.464956 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.465167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.464973 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.467691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.466537 4854 generic.go:356] "Generic (PLEG): container finished" podID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerID="a8079303242ca717008cc7a02c2fa23211a28027ea77f6dd36eaae08bd31f474" exitCode=0 Oct 31 03:19:43.467691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.466620 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" event={"ID":"bc8fd16b-51b4-47eb-ad4f-7226961f85f8","Type":"ContainerDied","Data":"a8079303242ca717008cc7a02c2fa23211a28027ea77f6dd36eaae08bd31f474"} Oct 31 03:19:43.467691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.467166 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs" pod "82cafdf5-57b6-46bd-9dd0-260613f4ccf2" (UID: "82cafdf5-57b6-46bd-9dd0-260613f4ccf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:43.469238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.469213 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb" pod "82cafdf5-57b6-46bd-9dd0-260613f4ccf2" (UID: "82cafdf5-57b6-46bd-9dd0-260613f4ccf2"). InnerVolumeSpecName "kube-api-access-6wzfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:43.469414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.469326 4854 generic.go:356] "Generic (PLEG): container finished" podID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerID="575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03" exitCode=0 Oct 31 03:19:43.469414 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.469374 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" Oct 31 03:19:43.469500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.469437 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" event={"ID":"82cafdf5-57b6-46bd-9dd0-260613f4ccf2","Type":"ContainerDied","Data":"575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03"} Oct 31 03:19:43.469500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.469468 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw" event={"ID":"82cafdf5-57b6-46bd-9dd0-260613f4ccf2","Type":"ContainerDied","Data":"9c19311992cb515f378d7466189bf1cd39851e2b6e5abe30ea0e85c5d65011c1"} Oct 31 03:19:43.469500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.469488 4854 scope.go:122] "RemoveContainer" containerID="575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03" Oct 31 03:19:43.471040 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.471017 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm" pod "82cafdf5-57b6-46bd-9dd0-260613f4ccf2" (UID: "82cafdf5-57b6-46bd-9dd0-260613f4ccf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.552489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.552437 4854 scope.go:122] "RemoveContainer" containerID="575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03" Oct 31 03:19:43.552928 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:19:43.552901 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03\": container with ID starting with 575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03 not found: ID does not exist" containerID="575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03" Oct 31 03:19:43.553030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.552940 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03"} err="failed to get container status \"575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03\": rpc error: code = NotFound desc = could not find container \"575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03\": container with ID starting with 575ee9d5f3e21bc4a1276dea206a844e090d9559d1cf1d88fc8187042dc43f03 not found: ID does not exist" Oct 31 03:19:43.555444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.555420 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:19:43.566439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566415 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache\") pod \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " Oct 31 03:19:43.566582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566565 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs\") pod \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " Oct 31 03:19:43.566658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566619 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28\" (UniqueName: \"kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28\") pod \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " Oct 31 03:19:43.566694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566661 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir\") pod \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " Oct 31 03:19:43.566694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566670 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache" pod "bc8fd16b-51b4-47eb-ad4f-7226961f85f8" (UID: "bc8fd16b-51b4-47eb-ad4f-7226961f85f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.566777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566737 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm\") pod \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " Oct 31 03:19:43.566981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566958 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir" pod "bc8fd16b-51b4-47eb-ad4f-7226961f85f8" (UID: "bc8fd16b-51b4-47eb-ad4f-7226961f85f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.567042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.566974 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home\") pod \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\" (UID: \"bc8fd16b-51b4-47eb-ad4f-7226961f85f8\") " Oct 31 03:19:43.567233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.567208 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home" pod "bc8fd16b-51b4-47eb-ad4f-7226961f85f8" (UID: "bc8fd16b-51b4-47eb-ad4f-7226961f85f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.568028 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.568002 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.568138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.568037 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.568138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.568052 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.568138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.568077 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wzfb\" (UniqueName: \"kubernetes.io/projected/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-kube-api-access-6wzfb\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.568138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.568091 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82cafdf5-57b6-46bd-9dd0-260613f4ccf2-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.568138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.568104 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.569120 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.569083 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm" pod "bc8fd16b-51b4-47eb-ad4f-7226961f85f8" (UID: "bc8fd16b-51b4-47eb-ad4f-7226961f85f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.569205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.569142 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28" pod "bc8fd16b-51b4-47eb-ad4f-7226961f85f8" (UID: "bc8fd16b-51b4-47eb-ad4f-7226961f85f8"). InnerVolumeSpecName "kube-api-access-q5t28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:43.569205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.569195 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs" pod "bc8fd16b-51b4-47eb-ad4f-7226961f85f8" (UID: "bc8fd16b-51b4-47eb-ad4f-7226961f85f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:43.670547 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.670510 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.670547 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.670545 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5t28\" (UniqueName: \"kubernetes.io/projected/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-kube-api-access-q5t28\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.670793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.670561 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc8fd16b-51b4-47eb-ad4f-7226961f85f8-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.855080 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.855040 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw"] Oct 31 03:19:43.859394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:43.859363 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-w0-edb9e6-kserve-b44bc6d94-hxlnw"] Oct 31 03:19:44.459493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:44.459418 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" path="/var/lib/kubelet/pods/82cafdf5-57b6-46bd-9dd0-260613f4ccf2/volumes" Oct 31 03:19:44.478916 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:44.478887 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" Oct 31 03:19:44.479046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:44.478924 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn" event={"ID":"bc8fd16b-51b4-47eb-ad4f-7226961f85f8","Type":"ContainerDied","Data":"e55640668fcc2dfc429b770a9365cb545d4c3759dca026a544e39dd93c2a668b"} Oct 31 03:19:44.479046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:44.478967 4854 scope.go:122] "RemoveContainer" containerID="a8079303242ca717008cc7a02c2fa23211a28027ea77f6dd36eaae08bd31f474" Oct 31 03:19:44.497753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:44.497723 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn"] Oct 31 03:19:44.501733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:44.501710 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/simulator-w0-edb9e6-kserve-69fb6678d6-mf9jn"] Oct 31 03:19:46.459577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:46.459540 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" path="/var/lib/kubelet/pods/bc8fd16b-51b4-47eb-ad4f-7226961f85f8/volumes" Oct 31 03:19:47.334438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.334218 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" podUID="96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" containerName="payload-processing" containerID="cri-o://d5680159cf5cbb661c9e415a4b433caf25a5c5825254f3e8fce98908b5c63936" gracePeriod=25 Oct 31 03:19:47.334733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.334691 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" podUID="40de8418-e312-44f2-b81b-b489af17a8b3" containerName="payload-pre-processing" containerID="cri-o://88877cc0bfb14263b678687223cff51d28f6c1c4553689376588074bd5edb976" gracePeriod=25 Oct 31 03:19:47.513290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.513044 4854 generic.go:356] "Generic (PLEG): container finished" podID="40de8418-e312-44f2-b81b-b489af17a8b3" containerID="88877cc0bfb14263b678687223cff51d28f6c1c4553689376588074bd5edb976" exitCode=0 Oct 31 03:19:47.513290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.513193 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" event={"ID":"40de8418-e312-44f2-b81b-b489af17a8b3","Type":"ContainerDied","Data":"88877cc0bfb14263b678687223cff51d28f6c1c4553689376588074bd5edb976"} Oct 31 03:19:47.516167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.515857 4854 generic.go:356] "Generic (PLEG): container finished" podID="96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" containerID="d5680159cf5cbb661c9e415a4b433caf25a5c5825254f3e8fce98908b5c63936" exitCode=0 Oct 31 03:19:47.516167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.516095 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" event={"ID":"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb","Type":"ContainerDied","Data":"d5680159cf5cbb661c9e415a4b433caf25a5c5825254f3e8fce98908b5c63936"} Oct 31 03:19:47.771340 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.771306 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:19:47.820779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.820755 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:19:47.844021 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.843986 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7\" (UniqueName: \"kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7\") pod \"40de8418-e312-44f2-b81b-b489af17a8b3\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " Oct 31 03:19:47.844613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.844587 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume\") pod \"40de8418-e312-44f2-b81b-b489af17a8b3\" (UID: \"40de8418-e312-44f2-b81b-b489af17a8b3\") " Oct 31 03:19:47.844747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.844670 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume\") pod \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " Oct 31 03:19:47.845142 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.845118 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume" pod "40de8418-e312-44f2-b81b-b489af17a8b3" (UID: "40de8418-e312-44f2-b81b-b489af17a8b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:47.845584 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.845530 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt\" (UniqueName: \"kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt\") pod \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\" (UID: \"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb\") " Oct 31 03:19:47.845868 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.845848 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume" pod "96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" (UID: "96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:47.846602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.846580 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40de8418-e312-44f2-b81b-b489af17a8b3-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:47.846700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.846609 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:47.848565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.848538 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7" pod "40de8418-e312-44f2-b81b-b489af17a8b3" (UID: "40de8418-e312-44f2-b81b-b489af17a8b3"). InnerVolumeSpecName "kube-api-access-6sdg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:47.851982 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.851955 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt" pod "96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" (UID: "96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb"). InnerVolumeSpecName "kube-api-access-rbrvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:47.948485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.948455 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbrvt\" (UniqueName: \"kubernetes.io/projected/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb-kube-api-access-rbrvt\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:47.948485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:47.948482 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdg7\" (UniqueName: \"kubernetes.io/projected/40de8418-e312-44f2-b81b-b489af17a8b3-kube-api-access-6sdg7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:48.589666 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.589634 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" Oct 31 03:19:48.590928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.590370 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk" event={"ID":"40de8418-e312-44f2-b81b-b489af17a8b3","Type":"ContainerDied","Data":"4f6d5f108d17c8a13af938939dd0128ec705ef459bb5376b8efb78fb293e8d0f"} Oct 31 03:19:48.590928 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.590475 4854 scope.go:122] "RemoveContainer" containerID="88877cc0bfb14263b678687223cff51d28f6c1c4553689376588074bd5edb976" Oct 31 03:19:48.594770 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.594088 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" event={"ID":"96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb","Type":"ContainerDied","Data":"1abab4fbc34222f2a16ea4625565c13efce982b07881bce660233d02d31b37c2"} Oct 31 03:19:48.594770 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.594188 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs" Oct 31 03:19:48.652660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.652636 4854 scope.go:122] "RemoveContainer" containerID="d5680159cf5cbb661c9e415a4b433caf25a5c5825254f3e8fce98908b5c63936" Oct 31 03:19:48.664109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.664082 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk"] Oct 31 03:19:48.679991 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.679954 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fd580-6cc45cfb59-dthjk"] Oct 31 03:19:48.686420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.686398 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs"] Oct 31 03:19:48.690682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:48.690658 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fd580-8f4bc955f-s86qs"] Oct 31 03:19:49.314381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.314351 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn"] Oct 31 03:19:49.315082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315066 4854 cpu_manager.go:414] "containerMap: removing container" podUID="96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" containerName="payload-processing" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315088 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" containerName="payload-processing" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315106 4854 cpu_manager.go:414] "containerMap: removing container" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315113 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315124 4854 cpu_manager.go:414] "containerMap: removing container" podUID="40de8418-e312-44f2-b81b-b489af17a8b3" containerName="payload-pre-processing" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315130 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40de8418-e312-44f2-b81b-b489af17a8b3" containerName="payload-pre-processing" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315140 4854 cpu_manager.go:414] "containerMap: removing container" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" Oct 31 03:19:49.315160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315145 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315192 4854 cpu_manager.go:414] "containerMap: removing container" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315199 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315210 4854 cpu_manager.go:414] "containerMap: removing container" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315217 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315232 4854 cpu_manager.go:414] "containerMap: removing container" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315239 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315272 4854 cpu_manager.go:414] "containerMap: removing container" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315279 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315445 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc9ef1f-7391-48a9-af37-186eaa30f981" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315458 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cafdf5-57b6-46bd-9dd0-260613f4ccf2" containerName="main" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315468 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8fd16b-51b4-47eb-ad4f-7226961f85f8" containerName="main" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315476 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="636c573f-b619-4775-984c-1f0ed67fb680" containerName="istio-proxy" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315485 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" containerName="payload-processing" Oct 31 03:19:49.315487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315493 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54e50f9-15a5-4a67-a11d-6d9c806699cc" containerName="istio-proxy" Oct 31 03:19:49.315892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315500 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="40de8418-e312-44f2-b81b-b489af17a8b3" containerName="payload-pre-processing" Oct 31 03:19:49.315892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.315507 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4aec8c4-40ac-4696-bdd5-ed3ec625e576" containerName="istio-proxy" Oct 31 03:19:49.319451 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.319407 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.322371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.322346 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b79c13-openshift-default-dockercfg-wfw7v\"" Oct 31 03:19:49.332437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.332419 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn"] Oct 31 03:19:49.383482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383452 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqvm\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.383646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383593 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.383717 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383651 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.383777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383751 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.383882 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383860 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.383997 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383902 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.383997 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.383935 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.384116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.384011 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.384116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.384093 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.487796 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.487903 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.487964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.487997 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.488075 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.488135 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.488412 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqvm\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.488542 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.488584 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.489806 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.497515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.490428 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.518284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.504006 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.518284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.504852 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.527334 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.526683 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.542272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.538962 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.548362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.541142 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.572301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.569050 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqvm\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.583274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.582645 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.636436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.636399 4854 generic.go:356] "Generic (PLEG): container finished" podID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerID="47d5d92bb5954d9f0a9926d125ca34b8911ee44a3be0c2e36df3af31af7df39b" exitCode=0 Oct 31 03:19:49.637558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.637088 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" event={"ID":"3b0b9870-c9ba-4005-8a97-417ca943412b","Type":"ContainerDied","Data":"47d5d92bb5954d9f0a9926d125ca34b8911ee44a3be0c2e36df3af31af7df39b"} Oct 31 03:19:49.638983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.638962 4854 scope.go:122] "RemoveContainer" containerID="47d5d92bb5954d9f0a9926d125ca34b8911ee44a3be0c2e36df3af31af7df39b" Oct 31 03:19:49.656301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.637373 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:49.669312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.658072 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:49.670285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.670226 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:49.670735 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:49.670620 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:50.160895 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.160863 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn"] Oct 31 03:19:50.161718 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:19:50.161675 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93ff988_5bc5_495c_99fa_2f8bdbe432b6.slice/crio-67adff6ed8293c3133a08cba8a2164f46e0075027ebb14c52f8a9cfb35c20f15 WatchSource:0}: Error finding container 67adff6ed8293c3133a08cba8a2164f46e0075027ebb14c52f8a9cfb35c20f15: Status 404 returned error can't find the container with id 67adff6ed8293c3133a08cba8a2164f46e0075027ebb14c52f8a9cfb35c20f15 Oct 31 03:19:50.165099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.165058 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:50.165214 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.165151 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:50.165284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.165218 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:19:50.219565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.219455 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2"] Oct 31 03:19:50.226011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.225993 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.240154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.239690 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2"] Oct 31 03:19:50.241599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.241579 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fccff5-serving-cert\"" Oct 31 03:19:50.325463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.325432 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.325626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.325487 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.325626 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.325609 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.325764 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.325736 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znm6r\" (UniqueName: \"kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.427723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.427518 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znm6r\" (UniqueName: \"kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.433705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.427867 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.433705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.427945 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.433705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.427994 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.433705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.429423 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.433705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.431973 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.433705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.432098 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.521486 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.521434 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40de8418-e312-44f2-b81b-b489af17a8b3" path="/var/lib/kubelet/pods/40de8418-e312-44f2-b81b-b489af17a8b3/volumes" Oct 31 03:19:50.526914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.526867 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb" path="/var/lib/kubelet/pods/96fd4cad-6df0-4c1c-aa28-c2b0a34c40bb/volumes" Oct 31 03:19:50.559763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.559715 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znm6r\" (UniqueName: \"kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r\") pod \"maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.592274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.591932 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:50.731874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.731790 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" event={"ID":"e93ff988-5bc5-495c-99fa-2f8bdbe432b6","Type":"ContainerStarted","Data":"ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465"} Oct 31 03:19:50.731874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.731844 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" event={"ID":"e93ff988-5bc5-495c-99fa-2f8bdbe432b6","Type":"ContainerStarted","Data":"67adff6ed8293c3133a08cba8a2164f46e0075027ebb14c52f8a9cfb35c20f15"} Oct 31 03:19:50.735947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.735922 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" event={"ID":"3b0b9870-c9ba-4005-8a97-417ca943412b","Type":"ContainerStarted","Data":"e69157fe1a9ff3b137965d87ade4d6f2db04050cc3055520cb798c8f4f2819ac"} Oct 31 03:19:50.762168 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:50.762119 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podStartSLOduration=1.7621044540000002 podStartE2EDuration="1.762104454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:50.753225597 +0000 UTC m=+1474.450392904" watchObservedRunningTime="2026-10-31 03:19:50.762104454 +0000 UTC m=+1474.459271758" Oct 31 03:19:51.089288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.088660 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2"] Oct 31 03:19:51.373151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.372923 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:19:51.373151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.373005 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:19:51.508487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.489301 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:19:51.508487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.506486 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:19:51.508487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.506538 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 03:19:51.638013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.637935 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:51.644367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.644340 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:51.746127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.746100 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" event={"ID":"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a","Type":"ContainerStarted","Data":"f6df01e58377e6cf5f91398223f593ade1b87071fd78177e41751d1cc35d2421"} Oct 31 03:19:51.746597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.746573 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:51.747508 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:51.747489 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:19:52.509859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:52.509815 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:19:52.510166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:52.510120 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 03:19:52.755861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:52.755821 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" event={"ID":"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a","Type":"ContainerStarted","Data":"556b5e115cf70a61dea79fd0c76cfdb61e52177eceed5574f0888e318f6d76f7"} Oct 31 03:19:52.756312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:52.755915 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:52.777993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:52.777913 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" podStartSLOduration=2.265451627 podStartE2EDuration="2.777901574s" totalImagesPullingTime="512.449947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:52.773633524 +0000 UTC m=+1476.470800829" watchObservedRunningTime="2026-10-31 03:19:52.777901574 +0000 UTC m=+1476.475068877" Oct 31 03:19:53.124124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:53.122535 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:19:53.494381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:53.493618 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:19:53.494381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:53.493688 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 03:19:54.491275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:54.490493 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:19:54.491275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:54.490556 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 03:19:55.498270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:55.497760 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:19:55.498270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:55.497831 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 03:19:56.503807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:56.503190 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:19:56.503807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:56.503286 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 03:19:57.494440 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:57.494412 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:19:57.495475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:57.495451 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:19:58.775399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.774856 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:19:58.817093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.817067 4854 generic.go:356] "Generic (PLEG): container finished" podID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerID="aad765d2c46678ea3f3b42ad4da24246d0d66383d2e0669de43fbac2f591adcc" exitCode=0 Oct 31 03:19:58.817393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.817370 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" event={"ID":"81f93acb-8038-4ca5-a28e-a6b275ce8760","Type":"ContainerDied","Data":"aad765d2c46678ea3f3b42ad4da24246d0d66383d2e0669de43fbac2f591adcc"} Oct 31 03:19:58.819295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.819232 4854 generic.go:356] "Generic (PLEG): container finished" podID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerID="e729d68c2cc60bdd87b1c5d3b6e0f4658803a8757c091ee3aad1972bfc11bf7d" exitCode=0 Oct 31 03:19:58.819463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.819442 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" event={"ID":"a76f6aae-6b69-4890-8556-20a3701f5f49","Type":"ContainerDied","Data":"e729d68c2cc60bdd87b1c5d3b6e0f4658803a8757c091ee3aad1972bfc11bf7d"} Oct 31 03:19:58.894554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.894527 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:19:58.980008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:58.978726 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.070529 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume\") pod \"a76f6aae-6b69-4890-8556-20a3701f5f49\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.070852 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume\") pod \"81f93acb-8038-4ca5-a28e-a6b275ce8760\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.070920 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs\" (UniqueName: \"kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs\") pod \"a76f6aae-6b69-4890-8556-20a3701f5f49\" (UID: \"a76f6aae-6b69-4890-8556-20a3701f5f49\") " Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.071752 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume" pod "a76f6aae-6b69-4890-8556-20a3701f5f49" (UID: "a76f6aae-6b69-4890-8556-20a3701f5f49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.072119 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume" pod "81f93acb-8038-4ca5-a28e-a6b275ce8760" (UID: "81f93acb-8038-4ca5-a28e-a6b275ce8760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.072473 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76f6aae-6b69-4890-8556-20a3701f5f49-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:59.073993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.072513 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f93acb-8038-4ca5-a28e-a6b275ce8760-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:59.075072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.075003 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs" pod "a76f6aae-6b69-4890-8556-20a3701f5f49" (UID: "a76f6aae-6b69-4890-8556-20a3701f5f49"). InnerVolumeSpecName "kube-api-access-m9wgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:59.173586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.173518 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q\" (UniqueName: \"kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q\") pod \"81f93acb-8038-4ca5-a28e-a6b275ce8760\" (UID: \"81f93acb-8038-4ca5-a28e-a6b275ce8760\") " Oct 31 03:19:59.174191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.174170 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9wgs\" (UniqueName: \"kubernetes.io/projected/a76f6aae-6b69-4890-8556-20a3701f5f49-kube-api-access-m9wgs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:59.175822 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.175799 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q" pod "81f93acb-8038-4ca5-a28e-a6b275ce8760" (UID: "81f93acb-8038-4ca5-a28e-a6b275ce8760"). InnerVolumeSpecName "kube-api-access-w6z6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:59.275319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.275290 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6z6q\" (UniqueName: \"kubernetes.io/projected/81f93acb-8038-4ca5-a28e-a6b275ce8760-kube-api-access-w6z6q\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:59.860336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.858370 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" event={"ID":"a76f6aae-6b69-4890-8556-20a3701f5f49","Type":"ContainerDied","Data":"3993c3ecacd6a9dfdf60accb28de9b34189afd8dcd712e1c869ec5fe880cc664"} Oct 31 03:19:59.860336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.858450 4854 scope.go:122] "RemoveContainer" containerID="e729d68c2cc60bdd87b1c5d3b6e0f4658803a8757c091ee3aad1972bfc11bf7d" Oct 31 03:19:59.860336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.858600 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9" Oct 31 03:19:59.865987 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.865959 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" event={"ID":"81f93acb-8038-4ca5-a28e-a6b275ce8760","Type":"ContainerDied","Data":"9c4c750bcd17e6709a756c5b61bf90d4a6a3637694b7dcef212ce0b26f3f2d37"} Oct 31 03:19:59.866105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.866055 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt" Oct 31 03:19:59.876630 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.876609 4854 scope.go:122] "RemoveContainer" containerID="aad765d2c46678ea3f3b42ad4da24246d0d66383d2e0669de43fbac2f591adcc" Oct 31 03:19:59.886962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.886942 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9"] Oct 31 03:19:59.893338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.893317 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-edb9e6-5fffc4975c-hgns9"] Oct 31 03:19:59.904648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.904627 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt"] Oct 31 03:19:59.914281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:19:59.910290 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-edb9e6-764fc668b7-jvcdt"] Oct 31 03:20:00.458876 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:00.458846 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" path="/var/lib/kubelet/pods/81f93acb-8038-4ca5-a28e-a6b275ce8760/volumes" Oct 31 03:20:00.459352 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:00.459337 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" path="/var/lib/kubelet/pods/a76f6aae-6b69-4890-8556-20a3701f5f49/volumes" Oct 31 03:20:08.551885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.551846 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t"] Oct 31 03:20:08.552976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.552953 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerName="payload-processing" Oct 31 03:20:08.553066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.552978 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerName="payload-processing" Oct 31 03:20:08.553066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.553047 4854 cpu_manager.go:414] "containerMap: removing container" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerName="payload-pre-processing" Oct 31 03:20:08.553066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.553055 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerName="payload-pre-processing" Oct 31 03:20:08.553235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.553220 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f93acb-8038-4ca5-a28e-a6b275ce8760" containerName="payload-pre-processing" Oct 31 03:20:08.553316 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.553239 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76f6aae-6b69-4890-8556-20a3701f5f49" containerName="payload-processing" Oct 31 03:20:08.558045 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.558017 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.572738 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.572706 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t"] Oct 31 03:20:08.576367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.576344 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b79c13-serving-cert\"" Oct 31 03:20:08.658960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.658928 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.659102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.658984 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6rr\" (UniqueName: \"kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.659337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.659312 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.659386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.659368 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.778300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.778211 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.786110 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.786072 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.786501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.786476 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6rr\" (UniqueName: \"kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.787235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.787209 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.808322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.791989 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.808322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.796176 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.820568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.820529 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.851939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.851900 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6rr\" (UniqueName: \"kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr\") pod \"maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:08.892786 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:08.892112 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:09.222267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:09.222173 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t"] Oct 31 03:20:09.226183 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:20:09.226157 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867896b1_871f_47e4_8a01_a283fe3e2810.slice/crio-58a24b2bd2c460b95a71afa5983b167029db0a2aa1c7d4c73259c7361f770fb0 WatchSource:0}: Error finding container 58a24b2bd2c460b95a71afa5983b167029db0a2aa1c7d4c73259c7361f770fb0: Status 404 returned error can't find the container with id 58a24b2bd2c460b95a71afa5983b167029db0a2aa1c7d4c73259c7361f770fb0 Oct 31 03:20:09.253804 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:20:09.253768 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:20:10.039429 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.039393 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" event={"ID":"867896b1-871f-47e4-8a01-a283fe3e2810","Type":"ContainerStarted","Data":"176ac9f3c15622e219eb01c016d7641d3bb3adc8ac09a8b17347f7c9690e4482"} Oct 31 03:20:10.039843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.039450 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" event={"ID":"867896b1-871f-47e4-8a01-a283fe3e2810","Type":"ContainerStarted","Data":"58a24b2bd2c460b95a71afa5983b167029db0a2aa1c7d4c73259c7361f770fb0"} Oct 31 03:20:10.039843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.039610 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:10.137275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.123418 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" podStartSLOduration=1.6853486659999999 podStartE2EDuration="2.123394848s" totalImagesPullingTime="438.046182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:10.085739403 +0000 UTC m=+1493.782906709" watchObservedRunningTime="2026-10-31 03:20:10.123394848 +0000 UTC m=+1493.820562153" Oct 31 03:20:10.365407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.365321 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb"] Oct 31 03:20:10.365914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.365877 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" containerID="cri-o://1fd05f3d1e63c2c6501fc8271722b948623a1e0c0e55c6d00a35a96e51017568" gracePeriod=30 Oct 31 03:20:10.399306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.398058 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr"] Oct 31 03:20:10.399306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.398542 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" containerID="cri-o://9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987" gracePeriod=30 Oct 31 03:20:10.415378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.412979 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b"] Oct 31 03:20:10.415378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:10.413329 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" containerID="cri-o://c70cefbb4c307f0c6edd48863131ef696f4022da3c672bd2fe4d9b7ac6b1404f" gracePeriod=30 Oct 31 03:20:12.871272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:12.870336 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:13.055923 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.055869 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:13.221336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.211532 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:20:13.272274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.271348 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:13.554275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.540358 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:13.879386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.877584 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:13.884273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.882596 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:20:13.988424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:13.959288 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.223331 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:14.222850 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:20:14.223513 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:14.223340 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:20:14.550349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:14.550299 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.677184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:14.677159 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:20:14.886770 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:14.886645 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:20:15.061290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:15.048578 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:20:15.061290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:15.061242 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:20:15.162138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:15.162054 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:20:15.341459 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:15.341413 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.412735 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:15.410840 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 03:20:15.948387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:15.930123 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:16.100270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.005229 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:16.100270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.084483 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:16.100546 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.100295 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:20:16.180274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.179394 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:16.208324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.196482 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:20:16.366485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.366423 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:20:16.368420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:16.368396 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:20:17.063298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.060982 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:20:17.166274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.120941 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:20:17.184417 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.140411 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:17.199229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.140455 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:17.200444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.199910 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:17.346755 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.346372 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:20:17.346755 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.346431 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:17.385621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:17.385571 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:18.093277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.091605 4854 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:20:18.093277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.091719 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 03:20:18.274274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.273393 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:20:18.346498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.282549 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:20:18.371278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.347617 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:20:18.667543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.367401 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:20:18.667543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.528610 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:20:18.667543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.644594 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:20:18.950428 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:20:18.938349 4854 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.441s" Oct 31 03:20:18.950428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.938473 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:20:18.950428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.938511 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:20:18.951138 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:18.951090 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt" podUID="c679a142-f04f-4b6f-b25e-ed237eb68f78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:19.059956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.059653 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wmfx5" podUID="c2b332bc-d96f-4af7-871e-be4a9a439e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:19.073612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.073564 4854 generic.go:356] "Generic (PLEG): container finished" podID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerID="1fd05f3d1e63c2c6501fc8271722b948623a1e0c0e55c6d00a35a96e51017568" exitCode=0 Oct 31 03:20:19.141848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.141816 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:20:19.154956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.154927 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:20:19.155127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.155074 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:20:19.155481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.155461 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:20:19.155936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.155506 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" event={"ID":"36f9e872-56dc-40ea-b858-f7212cc16ddc","Type":"ContainerDied","Data":"1fd05f3d1e63c2c6501fc8271722b948623a1e0c0e55c6d00a35a96e51017568"} Oct 31 03:20:19.157206 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.157154 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:20:19.163736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.161143 4854 generic.go:356] "Generic (PLEG): container finished" podID="8390e34b-2306-40bf-b38e-54da40de46b8" containerID="9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987" exitCode=0 Oct 31 03:20:19.163937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.163291 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" event={"ID":"8390e34b-2306-40bf-b38e-54da40de46b8","Type":"ContainerDied","Data":"9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987"} Oct 31 03:20:19.164096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.164011 4854 scope.go:122] "RemoveContainer" containerID="9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987" Oct 31 03:20:19.183594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.182771 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:20:19.206940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.204636 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283010 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283101 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283285 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283315 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283389 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283460 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283520 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283572 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.283701 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data\") pod \"8390e34b-2306-40bf-b38e-54da40de46b8\" (UID: \"8390e34b-2306-40bf-b38e-54da40de46b8\") " Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.292454 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.293047 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.293223 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.296026 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.296508 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.296726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.296683 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:19.312927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.312616 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.329558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.329514 4854 generic.go:356] "Generic (PLEG): container finished" podID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerID="c70cefbb4c307f0c6edd48863131ef696f4022da3c672bd2fe4d9b7ac6b1404f" exitCode=0 Oct 31 03:20:19.332101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.331050 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.335212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.335162 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" event={"ID":"1fa21fe4-1a2d-42b7-a3b8-da71071c2917","Type":"ContainerDied","Data":"c70cefbb4c307f0c6edd48863131ef696f4022da3c672bd2fe4d9b7ac6b1404f"} Oct 31 03:20:19.353445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.353412 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:19.353662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.353635 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm" pod "8390e34b-2306-40bf-b38e-54da40de46b8" (UID: "8390e34b-2306-40bf-b38e-54da40de46b8"). InnerVolumeSpecName "kube-api-access-lg4bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:19.387428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.387405 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.387580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.387566 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390e34b-2306-40bf-b38e-54da40de46b8-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.387672 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.387661 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.387753 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.387741 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4bm\" (UniqueName: \"kubernetes.io/projected/8390e34b-2306-40bf-b38e-54da40de46b8-kube-api-access-lg4bm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.387833 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.387822 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.387927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.387915 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390e34b-2306-40bf-b38e-54da40de46b8-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.388031 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.388019 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.388113 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.388102 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.388187 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.388177 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390e34b-2306-40bf-b38e-54da40de46b8-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.470355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.470339 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:20:19.474620 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:20:19.474589 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987\": container with ID starting with 9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987 not found: ID does not exist" containerID="9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987" Oct 31 03:20:19.474690 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:20:19.474656 4854 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987\": container with ID starting with 9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987 not found: ID does not exist" Oct 31 03:20:19.477438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.477412 4854 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987\": container with ID starting with 9ae9b3e800484e23bf85c54b7ff18d502ca2bcf6e208bc80c165688022970987 not found: ID does not exist" Oct 31 03:20:19.482392 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.482367 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:20:19.636123 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.636072 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.636408 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.636389 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.636639 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.636621 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.636773 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.636758 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.636900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.636886 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.637022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.637009 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.637155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.637141 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.637299 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.637282 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.637430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.637416 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.642341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.642314 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.642534 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.642514 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.642696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.642678 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.642841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.642825 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.643032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.643012 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.643165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.643149 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.643323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.643308 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.643458 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.643440 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket\") pod \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\" (UID: \"1fa21fe4-1a2d-42b7-a3b8-da71071c2917\") " Oct 31 03:20:19.652271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.643678 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket\") pod \"36f9e872-56dc-40ea-b858-f7212cc16ddc\" (UID: \"36f9e872-56dc-40ea-b858-f7212cc16ddc\") " Oct 31 03:20:19.653734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.653704 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.654113 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.654095 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.654560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.654540 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:19.655094 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.655073 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:19.693635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.693593 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.726320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.703551 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.726811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.726777 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.727005 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.726987 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.746496 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.746434 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.746728 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762743 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f9e872-56dc-40ea-b858-f7212cc16ddc-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762777 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762792 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762808 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762822 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762836 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762852 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762866 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762880 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.767275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.762898 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.811512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.811476 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:19.852983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.852947 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "kube-api-access-45c57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:19.854509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.854454 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:19.854857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.854818 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:19.855323 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.855296 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "kube-api-access-q8ftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:19.855608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.855578 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.855976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.855948 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy" pod "1fa21fe4-1a2d-42b7-a3b8-da71071c2917" (UID: "1fa21fe4-1a2d-42b7-a3b8-da71071c2917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:19.856685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.856665 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo" pod "36f9e872-56dc-40ea-b858-f7212cc16ddc" (UID: "36f9e872-56dc-40ea-b858-f7212cc16ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:19.864573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864540 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864572 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864586 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864601 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45c57\" (UniqueName: \"kubernetes.io/projected/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-kube-api-access-45c57\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864616 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ftr\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-kube-api-access-q8ftr\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864631 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864644 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f9e872-56dc-40ea-b858-f7212cc16ddc-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:19.864719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:19.864657 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa21fe4-1a2d-42b7-a3b8-da71071c2917-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:20.355677 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.355634 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" event={"ID":"1fa21fe4-1a2d-42b7-a3b8-da71071c2917","Type":"ContainerDied","Data":"1a89a07fda946a92f78dd00372e0b6cb300fe8a084ef2b1efda8c0720c26f887"} Oct 31 03:20:20.356163 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.355761 4854 scope.go:122] "RemoveContainer" containerID="c70cefbb4c307f0c6edd48863131ef696f4022da3c672bd2fe4d9b7ac6b1404f" Oct 31 03:20:20.356163 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.355936 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b" Oct 31 03:20:20.361562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.361540 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" Oct 31 03:20:20.363453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.363422 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" Oct 31 03:20:20.368525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.361578 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb" event={"ID":"36f9e872-56dc-40ea-b858-f7212cc16ddc","Type":"ContainerDied","Data":"e3855c18599b24f88cc7623276ae179f8657ffac0217c84d3ece8a3ed28875b7"} Oct 31 03:20:20.368701 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.368676 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr" event={"ID":"8390e34b-2306-40bf-b38e-54da40de46b8","Type":"ContainerDied","Data":"e1114d1ca829f6943fbc44c3152c55cf0d066d8ecb12cb83fe32548ce630d915"} Oct 31 03:20:20.372814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.372796 4854 scope.go:122] "RemoveContainer" containerID="1fd05f3d1e63c2c6501fc8271722b948623a1e0c0e55c6d00a35a96e51017568" Oct 31 03:20:20.447456 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.443330 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b"] Oct 31 03:20:20.465964 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.465928 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-fpd8b"] Oct 31 03:20:20.466213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.466174 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb"] Oct 31 03:20:20.469818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.469782 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-4kwzb"] Oct 31 03:20:20.552682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.552653 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr"] Oct 31 03:20:20.558027 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:20.557997 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-npntr"] Oct 31 03:20:20.560455 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:20:20.560418 4854 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390e34b_2306_40bf_b38e_54da40de46b8.slice/crio-e1114d1ca829f6943fbc44c3152c55cf0d066d8ecb12cb83fe32548ce630d915\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa21fe4_1a2d_42b7_a3b8_da71071c2917.slice/crio-1a89a07fda946a92f78dd00372e0b6cb300fe8a084ef2b1efda8c0720c26f887\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390e34b_2306_40bf_b38e_54da40de46b8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9e872_56dc_40ea_b858_f7212cc16ddc.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:20:22.507346 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:22.507320 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" path="/var/lib/kubelet/pods/1fa21fe4-1a2d-42b7-a3b8-da71071c2917/volumes" Oct 31 03:20:22.508657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:22.508632 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" path="/var/lib/kubelet/pods/36f9e872-56dc-40ea-b858-f7212cc16ddc/volumes" Oct 31 03:20:22.509298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:22.509281 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" path="/var/lib/kubelet/pods/8390e34b-2306-40bf-b38e-54da40de46b8/volumes" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.627850 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5"] Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.635903 4854 cpu_manager.go:414] "containerMap: removing container" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.635925 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.635953 4854 cpu_manager.go:414] "containerMap: removing container" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.635961 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.636023 4854 cpu_manager.go:414] "containerMap: removing container" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.636031 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.636179 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="8390e34b-2306-40bf-b38e-54da40de46b8" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.636192 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa21fe4-1a2d-42b7-a3b8-da71071c2917" containerName="istio-proxy" Oct 31 03:20:23.642172 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.636213 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f9e872-56dc-40ea-b858-f7212cc16ddc" containerName="istio-proxy" Oct 31 03:20:23.650498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.645738 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5"] Oct 31 03:20:23.650498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.645869 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.706978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.706936 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707162 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707038 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p6s\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707162 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707072 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707162 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707106 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707168 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707231 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707313 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707513 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707420 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.707513 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.707487 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.715275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.715238 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-62bf00-openshift-default-dockercfg-xf6pr\"" Oct 31 03:20:23.808320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808274 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808355 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808394 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808507 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808468 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808512 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808599 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808673 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p6s\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808703 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808755 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.808847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.808800 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.809042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.809017 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.809223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.809198 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.809464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.809396 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.809602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.809471 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.811290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.811264 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.812087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.812066 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.822488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.822465 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.825451 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.825427 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p6s\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:23.971552 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:23.971469 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:24.270668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.270636 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5"] Oct 31 03:20:24.272441 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:20:24.272401 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07765218_86e7_419d_8ae8_a981d9b47062.slice/crio-d11ecca8131feba441cee32f91334051454f218e70b0b4eefd0dc571a8c2a9c8 WatchSource:0}: Error finding container d11ecca8131feba441cee32f91334051454f218e70b0b4eefd0dc571a8c2a9c8: Status 404 returned error can't find the container with id d11ecca8131feba441cee32f91334051454f218e70b0b4eefd0dc571a8c2a9c8 Oct 31 03:20:24.276599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.276550 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:24.276774 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.276628 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:24.276774 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.276677 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:24.525954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.525873 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" event={"ID":"07765218-86e7-419d-8ae8-a981d9b47062","Type":"ContainerStarted","Data":"21160b265b36319842128e44bdaf0d1112f9e2926e86349ebed5aa6f5c4384b5"} Oct 31 03:20:24.525954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.525940 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" event={"ID":"07765218-86e7-419d-8ae8-a981d9b47062","Type":"ContainerStarted","Data":"d11ecca8131feba441cee32f91334051454f218e70b0b4eefd0dc571a8c2a9c8"} Oct 31 03:20:24.554678 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.554608 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podStartSLOduration=1.554589422 podStartE2EDuration="1.554589422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:24.546726851 +0000 UTC m=+1508.243894156" watchObservedRunningTime="2026-10-31 03:20:24.554589422 +0000 UTC m=+1508.251756725" Oct 31 03:20:24.972511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:24.972432 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:25.975324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:25.975181 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:25.975324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:25.975280 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:26.433690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.433643 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg"] Oct 31 03:20:26.438481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.438449 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.470119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.470077 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg"] Oct 31 03:20:26.531754 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.531721 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.531929 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.531783 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.531929 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.531922 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.532004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.531952 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.532079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.532055 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9mm\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.532143 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.532130 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.532281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.532268 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.532376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.532354 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.532431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.532415 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.542117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.542093 4854 generic.go:356] "Generic (PLEG): container finished" podID="a45e38e9-1553-4655-ae81-0875d884214e" containerID="6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c" exitCode=0 Oct 31 03:20:26.542304 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.542173 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" event={"ID":"a45e38e9-1553-4655-ae81-0875d884214e","Type":"ContainerDied","Data":"6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c"} Oct 31 03:20:26.542687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.542671 4854 scope.go:122] "RemoveContainer" containerID="6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c" Oct 31 03:20:26.543888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.543866 4854 generic.go:356] "Generic (PLEG): container finished" podID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerID="7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674" exitCode=0 Oct 31 03:20:26.543989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.543943 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" event={"ID":"78ebd920-0c0a-4cee-b2ef-8b307ce014f3","Type":"ContainerDied","Data":"7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674"} Oct 31 03:20:26.544608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.544567 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:26.544710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.544638 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:26.544710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.544653 4854 scope.go:122] "RemoveContainer" containerID="7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674" Oct 31 03:20:26.544710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.544675 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:26.546654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.546631 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:26.546759 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.546708 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:26.546759 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.546742 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.641920 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.641964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642028 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9mm\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642064 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642179 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642215 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642272 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642345 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.642393 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643854 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.643181 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.643854 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.643434 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.644004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.643984 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.644550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.644529 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.644705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.644687 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.647121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.647097 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.657356 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.657328 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9mm\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.660085 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.660056 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.661556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.661530 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.762704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.762678 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:26.975449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.974938 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:26.975449 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:26.974991 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:27.297363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.297317 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg"] Oct 31 03:20:27.335399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.335363 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:27.335546 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.335452 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:27.335546 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.335497 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:27.555498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.555408 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" event={"ID":"3597b530-55e8-43f0-ae5f-bc83519f7137","Type":"ContainerStarted","Data":"8b345ec6dd360cb3be84738699c7070c27647ff8a8a3868fb9da9cfdde6fcf58"} Oct 31 03:20:27.555498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.555471 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" event={"ID":"3597b530-55e8-43f0-ae5f-bc83519f7137","Type":"ContainerStarted","Data":"505c2d47e8e1b191ce3b38df6bdf8581ccc66675325e19244deefcd73dbcb241"} Oct 31 03:20:27.557451 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.557429 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" event={"ID":"a45e38e9-1553-4655-ae81-0875d884214e","Type":"ContainerStarted","Data":"bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170"} Oct 31 03:20:27.560676 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.560655 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" event={"ID":"78ebd920-0c0a-4cee-b2ef-8b307ce014f3","Type":"ContainerStarted","Data":"d6b29f5fc51f0a458355d302c09e93fa7ae99e5a6c06dab041276d371fe5870c"} Oct 31 03:20:27.579426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.579373 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podStartSLOduration=1.579360665 podStartE2EDuration="1.579360665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:27.577782361 +0000 UTC m=+1511.274949665" watchObservedRunningTime="2026-10-31 03:20:27.579360665 +0000 UTC m=+1511.276527969" Oct 31 03:20:27.767419 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.767375 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:28.006300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.991392 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.006300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:27.991458 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:28.062064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.053722 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:20:28.062064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.053863 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.062064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.053905 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:28.220669 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.220636 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:20:28.224815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.224364 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.224815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.224424 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:28.803711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.803670 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.804030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.803984 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:28.979490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.979459 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.979761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:28.979733 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:29.050717 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.050682 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.053275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.051287 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:29.121267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.121158 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r"] Oct 31 03:20:29.587274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.575385 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.587274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.575446 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:29.612819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.612780 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 31 03:20:29.772440 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.772401 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.772715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:29.772680 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:30.063618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.063583 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.064198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.064166 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:30.065154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.065126 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.065341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.065321 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:30.107277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.103987 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:20:30.250270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.247413 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:30.281274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.271031 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.281274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.273155 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:30.485332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.462335 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:20:30.551275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.538519 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:30.551275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.541959 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:20:30.595140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.594357 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 31 03:20:30.802166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.773719 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.802166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.773781 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:30.976275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.968423 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 03:20:30.976275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.971732 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.976275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:30.971778 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:31.110278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.110171 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:31.142403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.142360 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:31.142653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.113578 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 31 03:20:31.142820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.113519 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 03:20:31.259119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.236384 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:31.259119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.236449 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:31.344134 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.344082 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:31.384221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.381184 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:20:31.471084 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.471022 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:31.471837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.471814 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:20:31.608273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.583141 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:31.608273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.583682 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:20:31.678341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.651453 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 03:20:31.678341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.664564 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:20:31.818313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.816655 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:31.818313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.816741 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:31.977047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.968704 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:31.977047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:31.969376 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:20:32.012198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.008362 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:20:32.012198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.008425 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 03:20:32.090276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.085601 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:32.090276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.085670 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:32.156464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.147094 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:32.156464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.147607 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:20:32.336360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.335994 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:32.336360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.336067 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:32.533490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.453353 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:32.533490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.456789 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 03:20:32.578403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.578359 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:32.699324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.596503 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:20:32.699324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.667027 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:32.749810 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.749746 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:20:32.750132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.750112 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:20:32.776475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.772363 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:32.776475 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:32.772420 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:33.050889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.050854 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:33.051146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.051120 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:33.081870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.081840 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:20:33.120679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.120652 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:20:33.158314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.149946 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:33.179099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.160020 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:20:33.369652 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.369559 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:33.371793 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.371775 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:33.375174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.375152 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:20:33.376234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.376212 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:33.376428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.376400 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:33.385275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.382463 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:20:33.824277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.819973 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:33.824277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:33.820077 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:34.053659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.053615 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:34.053982 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.053941 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:34.125478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.125340 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" podUID="6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" containerName="payload-processing" containerID="cri-o://df7c81ca5e00b52de795527a2d72afd2b7b932f9a87f12c3cdd4e23deebcc09a" gracePeriod=25 Oct 31 03:20:34.224850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.224818 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:34.225426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.225396 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:34.769099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.768640 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:34.769099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:34.768700 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:35.096607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.082702 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:35.142443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.142400 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:35.242884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.182898 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:35.242884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.219205 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:20:35.242884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.219276 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 03:20:35.242884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.219729 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:20:35.319387 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.319352 4854 generic.go:356] "Generic (PLEG): container finished" podID="6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" containerID="df7c81ca5e00b52de795527a2d72afd2b7b932f9a87f12c3cdd4e23deebcc09a" exitCode=0 Oct 31 03:20:35.320131 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.320104 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" event={"ID":"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035","Type":"ContainerDied","Data":"df7c81ca5e00b52de795527a2d72afd2b7b932f9a87f12c3cdd4e23deebcc09a"} Oct 31 03:20:35.483663 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.483614 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:35.486953 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.486688 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:20:35.679021 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.678940 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume\") pod \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " Oct 31 03:20:35.679160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.679091 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k\" (UniqueName: \"kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k\") pod \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\" (UID: \"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035\") " Oct 31 03:20:35.679444 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.679407 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume" pod "6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" (UID: "6ec2d38e-dfd4-454a-93f0-9d3ab1aad035"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:35.680142 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.680114 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:35.682354 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.682328 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k" pod "6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" (UID: "6ec2d38e-dfd4-454a-93f0-9d3ab1aad035"). InnerVolumeSpecName "kube-api-access-bsp4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:35.770287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.767064 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:20:35.770287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.767146 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 03:20:35.783585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:35.782945 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsp4k\" (UniqueName: \"kubernetes.io/projected/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035-kube-api-access-bsp4k\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:36.056278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.055343 4854 patch_prober.go:29] interesting pod/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:20:36.056278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.055391 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 03:20:36.243576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.243530 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:20:36.246739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.246633 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:20:36.617306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.617231 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" Oct 31 03:20:36.621828 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.621776 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r" event={"ID":"6ec2d38e-dfd4-454a-93f0-9d3ab1aad035","Type":"ContainerDied","Data":"5c4d107d139ecd24e0cac504f508b1d382dc02916437e69921e79f7dbcdccc64"} Oct 31 03:20:36.622070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.622027 4854 scope.go:122] "RemoveContainer" containerID="df7c81ca5e00b52de795527a2d72afd2b7b932f9a87f12c3cdd4e23deebcc09a" Oct 31 03:20:36.652797 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.652769 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r"] Oct 31 03:20:36.656438 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.656409 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3b8712-d986b47ff-72q6r"] Oct 31 03:20:36.768379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.768351 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:36.768830 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.768817 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:36.769535 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.769511 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:20:36.919429 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.919351 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz"] Oct 31 03:20:36.920120 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.920100 4854 cpu_manager.go:414] "containerMap: removing container" podUID="6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" containerName="payload-processing" Oct 31 03:20:36.920215 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.920122 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" containerName="payload-processing" Oct 31 03:20:36.920367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.920352 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" containerName="payload-processing" Oct 31 03:20:36.926119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.926085 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:36.936870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.936836 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bbbc81-serving-cert\"" Oct 31 03:20:36.939224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:36.939201 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz"] Oct 31 03:20:37.050294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.050267 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:20:37.051368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.051348 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:20:37.068675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.068648 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.068778 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.068682 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.068869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.068851 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxx4j\" (UniqueName: \"kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.068915 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.068900 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.179831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.177300 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.179831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.177538 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.179831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.177578 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.179831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.177833 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxx4j\" (UniqueName: \"kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.187804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.186514 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.283273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.277350 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxx4j\" (UniqueName: \"kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.292864 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.284881 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.302116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.302084 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs\") pod \"maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:37.553356 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:37.553297 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:38.076894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:38.075940 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz"] Oct 31 03:20:38.082243 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:20:38.082099 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef60be2_e0aa_4791_90e6_e14cdf3dbb5d.slice/crio-16374ffb66008fd3463ce603ee8c6f7c3d7c0202fc9e802b957d065fb03dcf97 WatchSource:0}: Error finding container 16374ffb66008fd3463ce603ee8c6f7c3d7c0202fc9e802b957d065fb03dcf97: Status 404 returned error can't find the container with id 16374ffb66008fd3463ce603ee8c6f7c3d7c0202fc9e802b957d065fb03dcf97 Oct 31 03:20:38.458270 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:38.458170 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec2d38e-dfd4-454a-93f0-9d3ab1aad035" path="/var/lib/kubelet/pods/6ec2d38e-dfd4-454a-93f0-9d3ab1aad035/volumes" Oct 31 03:20:38.672735 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:38.672694 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" event={"ID":"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d","Type":"ContainerStarted","Data":"16374ffb66008fd3463ce603ee8c6f7c3d7c0202fc9e802b957d065fb03dcf97"} Oct 31 03:20:39.252405 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:20:39.252364 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:20:39.683727 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:39.683639 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" event={"ID":"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d","Type":"ContainerStarted","Data":"f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29"} Oct 31 03:20:39.684159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:39.683872 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:20:39.739178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:39.739115 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" podStartSLOduration=3.230971996 podStartE2EDuration="3.739093558s" totalImagesPullingTime="508.121562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:39.722695228 +0000 UTC m=+1523.419862532" watchObservedRunningTime="2026-10-31 03:20:39.739093558 +0000 UTC m=+1523.436260864" Oct 31 03:20:45.696210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:20:45.696179 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:21:08.129848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.129816 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s"] Oct 31 03:21:08.130365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.130141 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" containerID="cri-o://e69157fe1a9ff3b137965d87ade4d6f2db04050cc3055520cb798c8f4f2819ac" gracePeriod=30 Oct 31 03:21:08.620568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.620539 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts"] Oct 31 03:21:08.625708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.625688 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.644440 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.644409 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts"] Oct 31 03:21:08.651510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.651481 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8"] Oct 31 03:21:08.656974 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.656955 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.681614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.681587 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8"] Oct 31 03:21:08.728101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728058 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.728313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728113 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.728313 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728176 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728382 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728459 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728559 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728651 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728697 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728737 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728774 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728830 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728850 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728908 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728930 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728953 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrcb\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.728992 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.729023 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh7q\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.733300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.729066 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.836885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836566 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh7q\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.836885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836659 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.836885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836714 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.836885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836746 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.836885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836810 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.836885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836857 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836905 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.836957 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837018 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837079 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837141 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837186 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837268 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.837362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837295 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.837779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837387 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.837779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837421 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.837779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837459 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrcb\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.837779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.837518 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.839650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.838039 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.839650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.838691 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.839650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.838892 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.839650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.839231 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.850261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.843677 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.850261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.844614 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.850261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.845116 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.850261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.845710 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.850261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.849129 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.851430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.851366 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.859954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.859892 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh7q\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.867079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.866862 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.871761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.871689 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.908614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.891847 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.909621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.909356 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.924287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.918916 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.924287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.922318 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:08.925179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.925151 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrcb\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tllts\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.945261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.945217 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:08.976122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:08.976101 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:09.243533 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:21:09.243507 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54ac6d7_4e3c_4438_adaf_f0998762ad67.slice/crio-91eecdfecf9c7c45b0d6b16e832246fad98135b3ffffccff2ab1df9f6957c8be WatchSource:0}: Error finding container 91eecdfecf9c7c45b0d6b16e832246fad98135b3ffffccff2ab1df9f6957c8be: Status 404 returned error can't find the container with id 91eecdfecf9c7c45b0d6b16e832246fad98135b3ffffccff2ab1df9f6957c8be Oct 31 03:21:09.245005 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.244979 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts"] Oct 31 03:21:09.246751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.246709 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:09.246853 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.246821 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:09.246919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.246867 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:09.257648 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:09.257619 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:21:09.299854 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:21:09.299828 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb6194f_74ea_4818_bea2_fea075d97173.slice/crio-47d145a00e99ec341ee947208ce20989554ec136e33c490be478c815dc189e25 WatchSource:0}: Error finding container 47d145a00e99ec341ee947208ce20989554ec136e33c490be478c815dc189e25: Status 404 returned error can't find the container with id 47d145a00e99ec341ee947208ce20989554ec136e33c490be478c815dc189e25 Oct 31 03:21:09.300300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.300270 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8"] Oct 31 03:21:09.302903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.302874 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:09.302994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.302939 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:09.302994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:09.302969 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:10.253555 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.253509 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" event={"ID":"f54ac6d7-4e3c-4438-adaf-f0998762ad67","Type":"ContainerStarted","Data":"ef4c0973f287cae12e391cb03a452592fb75b5a37955124792272a2831cbf165"} Oct 31 03:21:10.254220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.254192 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" event={"ID":"f54ac6d7-4e3c-4438-adaf-f0998762ad67","Type":"ContainerStarted","Data":"91eecdfecf9c7c45b0d6b16e832246fad98135b3ffffccff2ab1df9f6957c8be"} Oct 31 03:21:10.257613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.256665 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" event={"ID":"3bb6194f-74ea-4818-bea2-fea075d97173","Type":"ContainerStarted","Data":"d3b136e60b5050065e1e140a0cc28fedbfbe33d48540a93d8ef43cbaf3b59d0e"} Oct 31 03:21:10.257613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.256709 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" event={"ID":"3bb6194f-74ea-4818-bea2-fea075d97173","Type":"ContainerStarted","Data":"47d145a00e99ec341ee947208ce20989554ec136e33c490be478c815dc189e25"} Oct 31 03:21:10.289076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.289030 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podStartSLOduration=2.289017575 podStartE2EDuration="2.289017575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:10.286202968 +0000 UTC m=+1553.983370284" watchObservedRunningTime="2026-10-31 03:21:10.289017575 +0000 UTC m=+1553.986184878" Oct 31 03:21:10.354477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.353494 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podStartSLOduration=2.353476958 podStartE2EDuration="2.353476958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:10.336795948 +0000 UTC m=+1554.033963255" watchObservedRunningTime="2026-10-31 03:21:10.353476958 +0000 UTC m=+1554.050644457" Oct 31 03:21:10.947278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.946298 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:10.949127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.949085 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tllts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:21:10.949296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.949145 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 03:21:10.976783 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.976754 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:10.977766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.977749 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:21:10.977865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:10.977785 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 03:21:11.955275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:11.946984 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tllts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:21:11.955275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:11.947066 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 03:21:11.990273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:11.981329 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:21:11.990273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:11.981382 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 03:21:12.533275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:12.526321 4854 patch_prober.go:29] interesting pod/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:21:12.533275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:12.526382 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:21:12.950320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:12.945908 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tllts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:21:12.950320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:12.945979 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 03:21:13.006308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.001310 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:21:13.006308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.001370 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 03:21:13.368150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.368116 4854 generic.go:356] "Generic (PLEG): container finished" podID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerID="e69157fe1a9ff3b137965d87ade4d6f2db04050cc3055520cb798c8f4f2819ac" exitCode=0 Oct 31 03:21:13.368333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.368174 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" event={"ID":"3b0b9870-c9ba-4005-8a97-417ca943412b","Type":"ContainerDied","Data":"e69157fe1a9ff3b137965d87ade4d6f2db04050cc3055520cb798c8f4f2819ac"} Oct 31 03:21:13.368333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.368219 4854 scope.go:122] "RemoveContainer" containerID="47d5d92bb5954d9f0a9926d125ca34b8911ee44a3be0c2e36df3af31af7df39b" Oct 31 03:21:13.652873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.652074 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:21:13.752273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.735867 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh"] Oct 31 03:21:13.759200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.759149 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.759406 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.759383 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.759616 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.764390 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.764504 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.759944 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.762221 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "kube-api-access-n8mrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.764743 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.765261 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.765286 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:13.765367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.765301 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mrz\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-kube-api-access-n8mrz\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:13.771242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.771211 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:13.772785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.772753 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:13.896329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.892050 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.896329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.892101 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.896329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.892228 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.896329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.892314 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo\") pod \"3b0b9870-c9ba-4005-8a97-417ca943412b\" (UID: \"3b0b9870-c9ba-4005-8a97-417ca943412b\") " Oct 31 03:21:13.896329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.893181 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:13.896329 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.893207 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:13.918218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.917864 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:13.918218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.918124 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:13.918218 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.918196 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:13.918721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.918381 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket" pod "3b0b9870-c9ba-4005-8a97-417ca943412b" (UID: "3b0b9870-c9ba-4005-8a97-417ca943412b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:13.951312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.946112 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tllts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:21:13.951312 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.946182 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 03:21:13.985281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.980617 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:21:13.985281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:13.980676 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 03:21:14.007273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.006923 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:14.007273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.006956 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b9870-c9ba-4005-8a97-417ca943412b-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:14.007273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.006972 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b9870-c9ba-4005-8a97-417ca943412b-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:14.007273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.006986 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b9870-c9ba-4005-8a97-417ca943412b-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:14.377032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.376997 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" event={"ID":"3b0b9870-c9ba-4005-8a97-417ca943412b","Type":"ContainerDied","Data":"5ca0490e31505d635953d13a37c1735c40c8ac7c7feda9c071a6ea92e5bc36ab"} Oct 31 03:21:14.377208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.377060 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s" Oct 31 03:21:14.377208 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.377084 4854 scope.go:122] "RemoveContainer" containerID="e69157fe1a9ff3b137965d87ade4d6f2db04050cc3055520cb798c8f4f2819ac" Oct 31 03:21:14.442332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.441443 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s"] Oct 31 03:21:14.446845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.446822 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3b8712-openshift-default-585bdb788f-s7w5s"] Oct 31 03:21:14.458696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.458670 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" path="/var/lib/kubelet/pods/3b0b9870-c9ba-4005-8a97-417ca943412b/volumes" Oct 31 03:21:14.951505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.950679 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tllts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:21:14.951505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.950783 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 03:21:14.979042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.979010 4854 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:21:14.979363 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:14.979334 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 03:21:15.950531 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:15.950503 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:15.951011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:15.950990 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:15.951832 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:15.951813 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:21:16.201980 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:16.201892 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:16.202965 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:16.202935 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:16.205062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:16.205044 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:21:28.786347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:28.786201 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" containerID="cri-o://2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b" gracePeriod=45 Oct 31 03:21:29.083430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.083398 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:21:29.176355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176326 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm\") pod \"49c944be-03b8-4015-9126-1eac4954734f\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " Oct 31 03:21:29.176548 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176525 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache\") pod \"49c944be-03b8-4015-9126-1eac4954734f\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " Oct 31 03:21:29.176686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176560 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2\" (UniqueName: \"kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2\") pod \"49c944be-03b8-4015-9126-1eac4954734f\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " Oct 31 03:21:29.176686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176599 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home\") pod \"49c944be-03b8-4015-9126-1eac4954734f\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " Oct 31 03:21:29.176807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176775 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir\") pod \"49c944be-03b8-4015-9126-1eac4954734f\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " Oct 31 03:21:29.176870 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176836 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs\" (UniqueName: \"kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs\") pod \"49c944be-03b8-4015-9126-1eac4954734f\" (UID: \"49c944be-03b8-4015-9126-1eac4954734f\") " Oct 31 03:21:29.176927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176889 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache" pod "49c944be-03b8-4015-9126-1eac4954734f" (UID: "49c944be-03b8-4015-9126-1eac4954734f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.176988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.176943 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home" pod "49c944be-03b8-4015-9126-1eac4954734f" (UID: "49c944be-03b8-4015-9126-1eac4954734f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.177099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.177079 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir" pod "49c944be-03b8-4015-9126-1eac4954734f" (UID: "49c944be-03b8-4015-9126-1eac4954734f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.177465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.177448 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.177551 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.177467 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.177551 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.177476 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.179217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.179190 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2" pod "49c944be-03b8-4015-9126-1eac4954734f" (UID: "49c944be-03b8-4015-9126-1eac4954734f"). InnerVolumeSpecName "kube-api-access-jfhc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:29.179390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.179299 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs" pod "49c944be-03b8-4015-9126-1eac4954734f" (UID: "49c944be-03b8-4015-9126-1eac4954734f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:29.179451 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.179403 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm" pod "49c944be-03b8-4015-9126-1eac4954734f" (UID: "49c944be-03b8-4015-9126-1eac4954734f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.294420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.291002 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhc2\" (UniqueName: \"kubernetes.io/projected/49c944be-03b8-4015-9126-1eac4954734f-kube-api-access-jfhc2\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.294420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.291046 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c944be-03b8-4015-9126-1eac4954734f-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.294420 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.291060 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c944be-03b8-4015-9126-1eac4954734f-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.541720 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.541687 4854 generic.go:356] "Generic (PLEG): container finished" podID="49c944be-03b8-4015-9126-1eac4954734f" containerID="2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b" exitCode=0 Oct 31 03:21:29.541924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.541731 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" Oct 31 03:21:29.541924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.541770 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" event={"ID":"49c944be-03b8-4015-9126-1eac4954734f","Type":"ContainerDied","Data":"2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b"} Oct 31 03:21:29.541924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.541816 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh" event={"ID":"49c944be-03b8-4015-9126-1eac4954734f","Type":"ContainerDied","Data":"7bd66895ad076a0b91fdfd1879d7c5515135165a586c8aaec239c7808e5d76fa"} Oct 31 03:21:29.541924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.541838 4854 scope.go:122] "RemoveContainer" containerID="2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b" Oct 31 03:21:29.553124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.553108 4854 scope.go:122] "RemoveContainer" containerID="2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b" Oct 31 03:21:29.553435 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:29.553416 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b\": container with ID starting with 2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b not found: ID does not exist" containerID="2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b" Oct 31 03:21:29.553512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.553446 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b"} err="failed to get container status \"2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b\": rpc error: code = NotFound desc = could not find container \"2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b\": container with ID starting with 2e9f385b019d2d34362617e4c5966cb13b4cb80221ed3fc083b5d6643903066b not found: ID does not exist" Oct 31 03:21:29.567306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.567281 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh"] Oct 31 03:21:29.570922 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.570897 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/premium-w3-99bc97-kserve-6974d5db44-kpvdh"] Oct 31 03:21:29.975184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.975099 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr"] Oct 31 03:21:29.975675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:29.975374 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" containerID="cri-o://fb259279580f2f9e0a24a084755b728bd857d9ae149bc2780d7c877684ec3792" gracePeriod=30 Oct 31 03:21:30.458176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:30.458144 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c944be-03b8-4015-9126-1eac4954734f" path="/var/lib/kubelet/pods/49c944be-03b8-4015-9126-1eac4954734f/volumes" Oct 31 03:21:34.358276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:34.357702 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:21:34.358276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:34.357786 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:21:35.176288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.170345 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:21:35.178538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.177678 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:21:35.318467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.318371 4854 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:21:35.473276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.457840 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:21:35.807277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.802574 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 03:21:35.897287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.895196 4854 generic.go:356] "Generic (PLEG): container finished" podID="82dcb70a-fe35-4891-97c4-86b113224244" containerID="fb259279580f2f9e0a24a084755b728bd857d9ae149bc2780d7c877684ec3792" exitCode=0 Oct 31 03:21:35.897287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.895441 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" event={"ID":"82dcb70a-fe35-4891-97c4-86b113224244","Type":"ContainerDied","Data":"fb259279580f2f9e0a24a084755b728bd857d9ae149bc2780d7c877684ec3792"} Oct 31 03:21:35.897287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:35.895506 4854 scope.go:122] "RemoveContainer" containerID="3acc7079f5e2ea72e968b98320f0b70893ba0d7ce7b6809b791ab29253a751fd" Oct 31 03:21:36.177375 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:36.177335 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:21:36.215288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:36.184961 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:21:36.253343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:36.240588 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2"] Oct 31 03:21:36.253343 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:36.240839 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" podUID="53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" containerName="maas-api" containerID="cri-o://556b5e115cf70a61dea79fd0c76cfdb61e52177eceed5574f0888e318f6d76f7" gracePeriod=30 Oct 31 03:21:36.864673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:36.864622 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 03:21:37.085275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.069627 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:21:37.170845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.170598 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" event={"ID":"82dcb70a-fe35-4891-97c4-86b113224244","Type":"ContainerDied","Data":"60e20a0465c00637193e4eb405eeebd4ce08ec1d19268b294e19b3065b1419f0"} Oct 31 03:21:37.170845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.170657 4854 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e20a0465c00637193e4eb405eeebd4ce08ec1d19268b294e19b3065b1419f0" Oct 31 03:21:37.171772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.171584 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:21:37.174342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.173897 4854 generic.go:356] "Generic (PLEG): container finished" podID="53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" containerID="556b5e115cf70a61dea79fd0c76cfdb61e52177eceed5574f0888e318f6d76f7" exitCode=0 Oct 31 03:21:37.175676 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.173918 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" event={"ID":"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a","Type":"ContainerDied","Data":"556b5e115cf70a61dea79fd0c76cfdb61e52177eceed5574f0888e318f6d76f7"} Oct 31 03:21:37.364694 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.364669 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:21:37.368972 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.368948 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369071 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.368993 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369071 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369033 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369150 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369084 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369241 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369190 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:37.369320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369285 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:37.369365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369316 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369407 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369463 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369437 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:37.369521 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369471 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369537 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369565 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token\") pod \"82dcb70a-fe35-4891-97c4-86b113224244\" (UID: \"82dcb70a-fe35-4891-97c4-86b113224244\") " Oct 31 03:21:37.369706 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369689 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:37.369854 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.369786 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:37.371032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.370628 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dcb70a-fe35-4891-97c4-86b113224244-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.371032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.370656 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.371032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.370672 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.371032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.370686 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.371032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.370699 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.371827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.371806 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:37.372015 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.371992 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "kube-api-access-ht5hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:37.372557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.372525 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:37.373763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.373746 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo" pod "82dcb70a-fe35-4891-97c4-86b113224244" (UID: "82dcb70a-fe35-4891-97c4-86b113224244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:37.471365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.471326 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r\" (UniqueName: \"kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r\") pod \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " Oct 31 03:21:37.471543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.471441 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca\") pod \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " Oct 31 03:21:37.471618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.471565 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs\") pod \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " Oct 31 03:21:37.471763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.471744 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls\") pod \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\" (UID: \"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a\") " Oct 31 03:21:37.471845 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.471772 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca" pod "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" (UID: "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:37.472379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.472356 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dcb70a-fe35-4891-97c4-86b113224244-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.472503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.472385 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.472503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.472400 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dcb70a-fe35-4891-97c4-86b113224244-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.472503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.472414 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5hd\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-kube-api-access-ht5hd\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.472503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.472428 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dcb70a-fe35-4891-97c4-86b113224244-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.474322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.474294 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls" pod "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" (UID: "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:37.474481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.474460 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r" pod "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" (UID: "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a"). InnerVolumeSpecName "kube-api-access-znm6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:37.474590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.474575 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs" pod "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" (UID: "53cfb2ba-7580-45f4-8bfb-2d0d79a3892a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:37.573581 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.573541 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.573581 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.573578 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:37.573581 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:37.573587 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znm6r\" (UniqueName: \"kubernetes.io/projected/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a-kube-api-access-znm6r\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:38.183009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.182980 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" Oct 31 03:21:38.183009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.182991 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2" event={"ID":"53cfb2ba-7580-45f4-8bfb-2d0d79a3892a","Type":"ContainerDied","Data":"f6df01e58377e6cf5f91398223f593ade1b87071fd78177e41751d1cc35d2421"} Oct 31 03:21:38.183576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.183076 4854 scope.go:122] "RemoveContainer" containerID="556b5e115cf70a61dea79fd0c76cfdb61e52177eceed5574f0888e318f6d76f7" Oct 31 03:21:38.183576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.183236 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr" Oct 31 03:21:38.227177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.227145 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr"] Oct 31 03:21:38.231399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.231376 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-wqnmr"] Oct 31 03:21:38.240129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.240108 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2"] Oct 31 03:21:38.243424 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.243405 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fccff5-7dc9cc54f4-76vx2"] Oct 31 03:21:38.459543 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.459473 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" path="/var/lib/kubelet/pods/53cfb2ba-7580-45f4-8bfb-2d0d79a3892a/volumes" Oct 31 03:21:38.459944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:38.459929 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dcb70a-fe35-4891-97c4-86b113224244" path="/var/lib/kubelet/pods/82dcb70a-fe35-4891-97c4-86b113224244/volumes" Oct 31 03:21:39.353282 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:39.347913 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:21:46.214394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:46.214360 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg"] Oct 31 03:21:46.215176 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:46.215140 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" containerID="cri-o://bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170" gracePeriod=30 Oct 31 03:21:48.552833 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:48.552800 4854 generic.go:356] "Generic (PLEG): container finished" podID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerID="ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465" exitCode=0 Oct 31 03:21:48.553279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:48.552872 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" event={"ID":"e93ff988-5bc5-495c-99fa-2f8bdbe432b6","Type":"ContainerDied","Data":"ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465"} Oct 31 03:21:48.553333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:48.553303 4854 scope.go:122] "RemoveContainer" containerID="ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465" Oct 31 03:21:48.554988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:48.554955 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:48.555108 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:48.555039 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:48.555108 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:48.555086 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:21:49.237588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:49.237555 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh"] Oct 31 03:21:49.237820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:49.237787 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" containerID="cri-o://4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40" gracePeriod=30 Oct 31 03:21:49.608345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:49.607645 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" event={"ID":"e93ff988-5bc5-495c-99fa-2f8bdbe432b6","Type":"ContainerStarted","Data":"b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754"} Oct 31 03:21:49.647306 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:49.645828 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:21:49.648532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:49.648414 4854 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:21:49.648532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:49.648488 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 03:21:50.638194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:50.638157 4854 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:21:50.638623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:50.638217 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 03:21:51.574468 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.574442 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:21:51.639151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.638726 4854 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:21:51.639151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.638798 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 03:21:51.649214 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.649174 4854 generic.go:356] "Generic (PLEG): container finished" podID="a45e38e9-1553-4655-ae81-0875d884214e" containerID="bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170" exitCode=0 Oct 31 03:21:51.649430 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.649407 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" event={"ID":"a45e38e9-1553-4655-ae81-0875d884214e","Type":"ContainerDied","Data":"bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170"} Oct 31 03:21:51.649514 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.649472 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" event={"ID":"a45e38e9-1553-4655-ae81-0875d884214e","Type":"ContainerDied","Data":"22d133c27e7010c35eeb0b59e5830e3fd5355784759a60a48077666635ca30f6"} Oct 31 03:21:51.649514 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.649504 4854 scope.go:122] "RemoveContainer" containerID="bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170" Oct 31 03:21:51.651135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.651110 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg" Oct 31 03:21:51.664472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.664422 4854 scope.go:122] "RemoveContainer" containerID="6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c" Oct 31 03:21:51.672815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.672789 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.672924 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.672878 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.672997 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.672921 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.672997 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.672960 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.673106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.672995 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.673106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.673070 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.673210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.673106 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.673210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.673163 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.673464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.673328 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy\") pod \"a45e38e9-1553-4655-ae81-0875d884214e\" (UID: \"a45e38e9-1553-4655-ae81-0875d884214e\") " Oct 31 03:21:51.674866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.674841 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:51.675217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.675190 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:51.675336 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.675278 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:51.675495 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.675470 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:51.675731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.675712 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:51.679152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.679056 4854 scope.go:122] "RemoveContainer" containerID="bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170" Oct 31 03:21:51.679740 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:51.679706 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170\": container with ID starting with bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170 not found: ID does not exist" containerID="bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170" Oct 31 03:21:51.679825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.679794 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170"} err="failed to get container status \"bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170\": rpc error: code = NotFound desc = could not find container \"bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170\": container with ID starting with bcf0ef522a8acb3e6c5b24b9717d36b026c2d2b233f9ef662a3e0b9b7d93c170 not found: ID does not exist" Oct 31 03:21:51.679902 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.679825 4854 scope.go:122] "RemoveContainer" containerID="6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c" Oct 31 03:21:51.679980 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.679708 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:51.681405 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:51.681227 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c\": container with ID starting with 6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c not found: ID does not exist" containerID="6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c" Oct 31 03:21:51.681405 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.681374 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c"} err="failed to get container status \"6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c\": rpc error: code = NotFound desc = could not find container \"6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c\": container with ID starting with 6984ad15d60062a66852106b481b8e3460013eaf70986c909d5696333a609f5c not found: ID does not exist" Oct 31 03:21:51.682163 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.682137 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:51.682415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.682389 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "kube-api-access-ph9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:51.682795 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.682768 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token" pod "a45e38e9-1553-4655-ae81-0875d884214e" (UID: "a45e38e9-1553-4655-ae81-0875d884214e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:51.746618 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.746591 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:21:51.775143 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775116 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9ms\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-kube-api-access-ph9ms\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775143 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a45e38e9-1553-4655-ae81-0875d884214e-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775159 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a45e38e9-1553-4655-ae81-0875d884214e-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775172 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775184 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775197 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775209 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a45e38e9-1553-4655-ae81-0875d884214e-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775221 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:51.775300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:51.775233 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a45e38e9-1553-4655-ae81-0875d884214e-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:52.203055 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.202437 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg"] Oct 31 03:21:52.262029 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.261994 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fccff5-openshift-default-67bd488879-gvgbg"] Oct 31 03:21:52.458462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.458385 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45e38e9-1553-4655-ae81-0875d884214e" path="/var/lib/kubelet/pods/a45e38e9-1553-4655-ae81-0875d884214e/volumes" Oct 31 03:21:52.640274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.639387 4854 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:21:52.640274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.639459 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 03:21:52.651427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.651397 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:21:52.672767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:52.672730 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:21:53.638201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:53.638169 4854 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:21:53.638399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:53.638232 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 03:21:53.757423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:53.751082 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf"] Oct 31 03:21:53.757423 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:53.751713 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" containerID="cri-o://3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a" gracePeriod=30 Oct 31 03:21:53.780489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:53.780422 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs"] Oct 31 03:21:53.780961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:53.780895 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" containerID="cri-o://4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c" gracePeriod=30 Oct 31 03:21:54.563966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.563943 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:21:54.567023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.566995 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567041 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567164 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567105 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567211 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567336 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567496 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567451 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:54.567568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567479 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:54.567568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567465 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567506 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:54.567736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567565 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567610 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567628 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:54.567736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567639 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo\") pod \"91f59594-de06-478d-a297-6a15040a7353\" (UID: \"91f59594-de06-478d-a297-6a15040a7353\") " Oct 31 03:21:54.567936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.567837 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:54.568573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.568547 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.568696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.568579 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.568696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.568596 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f59594-de06-478d-a297-6a15040a7353-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.568696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.568610 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.568696 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.568623 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.569721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.569700 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:54.569836 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.569815 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:54.569911 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.569841 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "kube-api-access-4gvgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:54.570092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.570071 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo" pod "91f59594-de06-478d-a297-6a15040a7353" (UID: "91f59594-de06-478d-a297-6a15040a7353"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:54.638510 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.638476 4854 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:21:54.638673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.638533 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 03:21:54.669900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.669864 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvgz\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-kube-api-access-4gvgz\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.669900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.669894 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f59594-de06-478d-a297-6a15040a7353-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.669900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.669904 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f59594-de06-478d-a297-6a15040a7353-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.670128 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.669913 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f59594-de06-478d-a297-6a15040a7353-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:54.687573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.687549 4854 generic.go:356] "Generic (PLEG): container finished" podID="91f59594-de06-478d-a297-6a15040a7353" containerID="4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40" exitCode=0 Oct 31 03:21:54.687672 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.687631 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" Oct 31 03:21:54.687672 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.687648 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" event={"ID":"91f59594-de06-478d-a297-6a15040a7353","Type":"ContainerDied","Data":"4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40"} Oct 31 03:21:54.687806 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.687716 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh" event={"ID":"91f59594-de06-478d-a297-6a15040a7353","Type":"ContainerDied","Data":"787cfa41733a583c8358a34112cd41abbd1add62182cffce290972cdf49d4a54"} Oct 31 03:21:54.687806 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.687750 4854 scope.go:122] "RemoveContainer" containerID="4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40" Oct 31 03:21:54.725082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.725062 4854 scope.go:122] "RemoveContainer" containerID="4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620" Oct 31 03:21:54.736999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.736972 4854 scope.go:122] "RemoveContainer" containerID="4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40" Oct 31 03:21:54.737440 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:54.737394 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40\": container with ID starting with 4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40 not found: ID does not exist" containerID="4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40" Oct 31 03:21:54.737531 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.737430 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40"} err="failed to get container status \"4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40\": rpc error: code = NotFound desc = could not find container \"4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40\": container with ID starting with 4469bf48d9e57e86791d4bab68f8430fa9bbb39e3f8d4447ba9eb4956e3f7e40 not found: ID does not exist" Oct 31 03:21:54.737531 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.737453 4854 scope.go:122] "RemoveContainer" containerID="4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620" Oct 31 03:21:54.737815 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:54.737779 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620\": container with ID starting with 4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620 not found: ID does not exist" containerID="4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620" Oct 31 03:21:54.737888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.737826 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620"} err="failed to get container status \"4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620\": rpc error: code = NotFound desc = could not find container \"4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620\": container with ID starting with 4afcb05487b85c4ec4c942bae565a46eed9226a81fefa50e8fa9c468e831d620 not found: ID does not exist" Oct 31 03:21:54.738498 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.738478 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh"] Oct 31 03:21:54.743518 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:54.743499 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fd2e0b-openshift-default-675ddcc9b5-pd8fh"] Oct 31 03:21:55.766041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:55.766010 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:21:55.767585 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:55.767565 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:21:56.458020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:56.457989 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f59594-de06-478d-a297-6a15040a7353" path="/var/lib/kubelet/pods/91f59594-de06-478d-a297-6a15040a7353/volumes" Oct 31 03:21:59.070433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.070404 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:21:59.171958 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.171930 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.171998 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172051 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172083 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172261 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172129 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172239 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172242 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.172435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172321 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172357 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172424 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy\") pod \"db86aec6-406a-44f5-b7a6-01876224e495\" (UID: \"db86aec6-406a-44f5-b7a6-01876224e495\") " Oct 31 03:21:59.172566 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172419 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.172612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172437 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.172671 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172627 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.172892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.172860 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:59.173035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.173018 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.173105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.173039 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.173105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.173049 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db86aec6-406a-44f5-b7a6-01876224e495-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.173105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.173059 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.173105 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.173074 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.174905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.174876 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:59.174905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.174901 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.175122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.175102 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.175182 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.175115 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj" pod "db86aec6-406a-44f5-b7a6-01876224e495" (UID: "db86aec6-406a-44f5-b7a6-01876224e495"). InnerVolumeSpecName "kube-api-access-446fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.274586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.274549 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db86aec6-406a-44f5-b7a6-01876224e495-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.274586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.274587 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-446fj\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-kube-api-access-446fj\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.274994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.274601 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db86aec6-406a-44f5-b7a6-01876224e495-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.274994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.274614 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db86aec6-406a-44f5-b7a6-01876224e495-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.432428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.432373 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:21:59.477511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477481 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.477687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477533 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.477687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477588 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.477837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477698 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.477837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477728 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.477837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477797 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.478037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477837 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.478037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477924 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.478037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.477997 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.478037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.478009 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.478297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.478110 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket\") pod \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\" (UID: \"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2\") " Oct 31 03:21:59.479033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.479008 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.479139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.479036 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.479213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.479192 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.479539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.479511 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:59.479867 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.479830 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.482326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.481526 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.482326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.481944 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.482326 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.482169 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "kube-api-access-wxkvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.486385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.486365 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo" pod "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" (UID: "8e8d3bba-a73b-4ac8-a0bc-4b56415346f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586861 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586900 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586917 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586931 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxkvp\" (UniqueName: \"kubernetes.io/projected/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-kube-api-access-wxkvp\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586946 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586959 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.591273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.586973 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.783241 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.783211 4854 generic.go:356] "Generic (PLEG): container finished" podID="db86aec6-406a-44f5-b7a6-01876224e495" containerID="4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c" exitCode=0 Oct 31 03:21:59.783434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.783300 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" Oct 31 03:21:59.783434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.783319 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" event={"ID":"db86aec6-406a-44f5-b7a6-01876224e495","Type":"ContainerDied","Data":"4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c"} Oct 31 03:21:59.783434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.783383 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs" event={"ID":"db86aec6-406a-44f5-b7a6-01876224e495","Type":"ContainerDied","Data":"db346bbbec07dbd17797fb8659aad032568105935755a559e3ee6a4428db5d66"} Oct 31 03:21:59.783434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.783413 4854 scope.go:122] "RemoveContainer" containerID="4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c" Oct 31 03:21:59.785229 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.785201 4854 generic.go:356] "Generic (PLEG): container finished" podID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerID="3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a" exitCode=0 Oct 31 03:21:59.785360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.785291 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" event={"ID":"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2","Type":"ContainerDied","Data":"3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a"} Oct 31 03:21:59.785360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.785331 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" event={"ID":"8e8d3bba-a73b-4ac8-a0bc-4b56415346f2","Type":"ContainerDied","Data":"6aae9c198194f5576ec71ebc2019fb365c0fa5423e9b1ae1bd6adbf99c4e2b09"} Oct 31 03:21:59.785492 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.785398 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf" Oct 31 03:21:59.796368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.796345 4854 scope.go:122] "RemoveContainer" containerID="4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c" Oct 31 03:21:59.796688 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:59.796652 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c\": container with ID starting with 4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c not found: ID does not exist" containerID="4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c" Oct 31 03:21:59.796897 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.796693 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c"} err="failed to get container status \"4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c\": rpc error: code = NotFound desc = could not find container \"4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c\": container with ID starting with 4c01c11ba05541ad84c7546357e87a38babd1fc867c3a994e3b622fb6151dd3c not found: ID does not exist" Oct 31 03:21:59.796897 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.796710 4854 scope.go:122] "RemoveContainer" containerID="3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a" Oct 31 03:21:59.807905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.807887 4854 scope.go:122] "RemoveContainer" containerID="3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a" Oct 31 03:21:59.808165 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:21:59.808143 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a\": container with ID starting with 3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a not found: ID does not exist" containerID="3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a" Oct 31 03:21:59.808268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.808174 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a"} err="failed to get container status \"3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a\": rpc error: code = NotFound desc = could not find container \"3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a\": container with ID starting with 3f524fbbf849011de0defc3dc9343362c08fed069f5be4a2bf2b9515ec1b1e9a not found: ID does not exist" Oct 31 03:21:59.815831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.815810 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf"] Oct 31 03:21:59.822282 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.822261 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-lpsbf"] Oct 31 03:21:59.831502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.831480 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs"] Oct 31 03:21:59.837119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:21:59.837101 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-45dhs"] Oct 31 03:22:00.504118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:00.504092 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" path="/var/lib/kubelet/pods/8e8d3bba-a73b-4ac8-a0bc-4b56415346f2/volumes" Oct 31 03:22:00.505012 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:00.504987 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db86aec6-406a-44f5-b7a6-01876224e495" path="/var/lib/kubelet/pods/db86aec6-406a-44f5-b7a6-01876224e495/volumes" Oct 31 03:22:03.989083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:03.989050 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw"] Oct 31 03:22:03.989617 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:03.989347 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" podUID="93aa48af-74e0-458d-8090-2cb2b031f371" containerName="maas-api" containerID="cri-o://3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5" gracePeriod=30 Oct 31 03:22:04.523493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.523453 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:22:04.573488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.573439 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls\") pod \"93aa48af-74e0-458d-8090-2cb2b031f371\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " Oct 31 03:22:04.582760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.582725 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d\" (UniqueName: \"kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d\") pod \"93aa48af-74e0-458d-8090-2cb2b031f371\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " Oct 31 03:22:04.582965 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.582948 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs\") pod \"93aa48af-74e0-458d-8090-2cb2b031f371\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " Oct 31 03:22:04.583096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.583081 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca\") pod \"93aa48af-74e0-458d-8090-2cb2b031f371\" (UID: \"93aa48af-74e0-458d-8090-2cb2b031f371\") " Oct 31 03:22:04.587777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.587748 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca" pod "93aa48af-74e0-458d-8090-2cb2b031f371" (UID: "93aa48af-74e0-458d-8090-2cb2b031f371"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:04.604679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.604413 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls" pod "93aa48af-74e0-458d-8090-2cb2b031f371" (UID: "93aa48af-74e0-458d-8090-2cb2b031f371"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:04.604679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.604438 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs" pod "93aa48af-74e0-458d-8090-2cb2b031f371" (UID: "93aa48af-74e0-458d-8090-2cb2b031f371"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:04.606515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.606130 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d" pod "93aa48af-74e0-458d-8090-2cb2b031f371" (UID: "93aa48af-74e0-458d-8090-2cb2b031f371"). InnerVolumeSpecName "kube-api-access-wrx8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:04.686215 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.686175 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrx8d\" (UniqueName: \"kubernetes.io/projected/93aa48af-74e0-458d-8090-2cb2b031f371-kube-api-access-wrx8d\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:04.686215 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.686218 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:04.686441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.686233 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa48af-74e0-458d-8090-2cb2b031f371-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:04.686441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.686264 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa48af-74e0-458d-8090-2cb2b031f371-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:04.961879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.961844 4854 generic.go:356] "Generic (PLEG): container finished" podID="93aa48af-74e0-458d-8090-2cb2b031f371" containerID="3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5" exitCode=0 Oct 31 03:22:04.962124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.962105 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" event={"ID":"93aa48af-74e0-458d-8090-2cb2b031f371","Type":"ContainerDied","Data":"3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5"} Oct 31 03:22:04.962280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.962244 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" event={"ID":"93aa48af-74e0-458d-8090-2cb2b031f371","Type":"ContainerDied","Data":"5eaeafcdf8d19089472d735863431f751b16c9766efdc4f7aa19675332c0f7a9"} Oct 31 03:22:04.962398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.962385 4854 scope.go:122] "RemoveContainer" containerID="3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5" Oct 31 03:22:04.962658 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:04.962642 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw" Oct 31 03:22:05.059192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:05.059171 4854 scope.go:122] "RemoveContainer" containerID="3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5" Oct 31 03:22:05.061310 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:05.060609 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5\": container with ID starting with 3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5 not found: ID does not exist" containerID="3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5" Oct 31 03:22:05.061310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:05.060652 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5"} err="failed to get container status \"3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5\": rpc error: code = NotFound desc = could not find container \"3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5\": container with ID starting with 3551075ca0db2f942d70ae387aeb86b3eef4a89655cc45e571d38ac42d2eaad5 not found: ID does not exist" Oct 31 03:22:05.071523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:05.071499 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw"] Oct 31 03:22:05.074759 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:05.074737 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-677281-b8c8686f4-ms8rw"] Oct 31 03:22:06.458196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:06.458160 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93aa48af-74e0-458d-8090-2cb2b031f371" path="/var/lib/kubelet/pods/93aa48af-74e0-458d-8090-2cb2b031f371/volumes" Oct 31 03:22:09.199976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.199936 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc"] Oct 31 03:22:09.200863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200840 4854 cpu_manager.go:414] "containerMap: removing container" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200864 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200885 4854 cpu_manager.go:414] "containerMap: removing container" podUID="93aa48af-74e0-458d-8090-2cb2b031f371" containerName="maas-api" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200891 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aa48af-74e0-458d-8090-2cb2b031f371" containerName="maas-api" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200908 4854 cpu_manager.go:414] "containerMap: removing container" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200915 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200929 4854 cpu_manager.go:414] "containerMap: removing container" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200936 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200950 4854 cpu_manager.go:414] "containerMap: removing container" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" Oct 31 03:22:09.200967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200956 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200974 4854 cpu_manager.go:414] "containerMap: removing container" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.200982 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201004 4854 cpu_manager.go:414] "containerMap: removing container" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201013 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201043 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201050 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201064 4854 cpu_manager.go:414] "containerMap: removing container" podUID="53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" containerName="maas-api" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201072 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" containerName="maas-api" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201083 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201090 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201105 4854 cpu_manager.go:414] "containerMap: removing container" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201113 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201126 4854 cpu_manager.go:414] "containerMap: removing container" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201133 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201142 4854 cpu_manager.go:414] "containerMap: removing container" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201149 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201318 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201333 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cfb2ba-7580-45f4-8bfb-2d0d79a3892a" containerName="maas-api" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201345 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45e38e9-1553-4655-ae81-0875d884214e" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201356 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201365 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201374 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0b9870-c9ba-4005-8a97-417ca943412b" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201387 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="db86aec6-406a-44f5-b7a6-01876224e495" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201396 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aa48af-74e0-458d-8090-2cb2b031f371" containerName="maas-api" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201405 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201413 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8d3bba-a73b-4ac8-a0bc-4b56415346f2" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201428 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dcb70a-fe35-4891-97c4-86b113224244" containerName="istio-proxy" Oct 31 03:22:09.201453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.201439 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c944be-03b8-4015-9126-1eac4954734f" containerName="main" Oct 31 03:22:09.205656 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.205634 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.210173 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.210151 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-677281-openshift-default-dockercfg-2vqqr\"" Oct 31 03:22:09.221070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.221044 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn"] Oct 31 03:22:09.222328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.222232 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f59594-de06-478d-a297-6a15040a7353" containerName="istio-proxy" Oct 31 03:22:09.226737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.226717 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc"] Oct 31 03:22:09.226861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.226824 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.264733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.264705 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn"] Oct 31 03:22:09.267295 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:09.265734 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:22:09.283685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283653 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.283861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283744 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.283861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283789 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.283861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283828 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.283861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283851 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzm2t\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284111 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283906 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.284111 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.283943 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284111 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284054 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.284234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284118 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd46x\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.284234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284157 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.284234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284187 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284210 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284273 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.284441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284305 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284375 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284423 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.284612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284456 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.284612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.284493 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.390280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.388215 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd46x\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.390280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.388302 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406551 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406608 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406640 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406686 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406713 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406843 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406893 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406924 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.406964 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407019 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407117 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407155 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407194 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407222 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzm2t\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407317 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407360 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.407511 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.408030 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.409135 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.413278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.412360 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.415875 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd46x\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.416194 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.419144 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.419477 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.420062 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.424594 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.425271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.424825 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.428126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.426434 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.428126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.426747 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.434943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.430650 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.434943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.433043 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.434943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.433177 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.434943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.434564 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzm2t\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.435982 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.435945 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.521611 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.521590 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:09.564957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.564932 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:09.810494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.810470 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc"] Oct 31 03:22:09.811638 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:22:09.811611 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492d17b7_7dca_48ca_991e_62e32e839673.slice/crio-8483cfeb172b35afbb1c1387fbac2e6080a02d4196bb331c7f47e2155dacd2d4 WatchSource:0}: Error finding container 8483cfeb172b35afbb1c1387fbac2e6080a02d4196bb331c7f47e2155dacd2d4: Status 404 returned error can't find the container with id 8483cfeb172b35afbb1c1387fbac2e6080a02d4196bb331c7f47e2155dacd2d4 Oct 31 03:22:09.814281 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.814216 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.814393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.814304 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.814393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.814350 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.874825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.874801 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn"] Oct 31 03:22:09.875813 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:22:09.875780 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a112cb_cb85_47b4_b21c_dcef4e92ac61.slice/crio-c21ada3bbe36588cf9fa0ac1efd9339c8f9f5869a68f4747667bd0e358b586a2 WatchSource:0}: Error finding container c21ada3bbe36588cf9fa0ac1efd9339c8f9f5869a68f4747667bd0e358b586a2: Status 404 returned error can't find the container with id c21ada3bbe36588cf9fa0ac1efd9339c8f9f5869a68f4747667bd0e358b586a2 Oct 31 03:22:09.882935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.882848 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.882935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.882925 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.882935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:09.882958 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:10.018102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.017593 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" event={"ID":"03a112cb-cb85-47b4-b21c-dcef4e92ac61","Type":"ContainerStarted","Data":"8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0"} Oct 31 03:22:10.018102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.017654 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" event={"ID":"03a112cb-cb85-47b4-b21c-dcef4e92ac61","Type":"ContainerStarted","Data":"c21ada3bbe36588cf9fa0ac1efd9339c8f9f5869a68f4747667bd0e358b586a2"} Oct 31 03:22:10.020389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.020179 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" event={"ID":"492d17b7-7dca-48ca-991e-62e32e839673","Type":"ContainerStarted","Data":"cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a"} Oct 31 03:22:10.020389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.020219 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" event={"ID":"492d17b7-7dca-48ca-991e-62e32e839673","Type":"ContainerStarted","Data":"8483cfeb172b35afbb1c1387fbac2e6080a02d4196bb331c7f47e2155dacd2d4"} Oct 31 03:22:10.043539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.043479 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podStartSLOduration=1.043466173 podStartE2EDuration="1.043466173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:10.041406027 +0000 UTC m=+1613.738573331" watchObservedRunningTime="2026-10-31 03:22:10.043466173 +0000 UTC m=+1613.740633477" Oct 31 03:22:10.072288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.072175 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podStartSLOduration=1.072160534 podStartE2EDuration="1.072160534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:10.069226477 +0000 UTC m=+1613.766393775" watchObservedRunningTime="2026-10-31 03:22:10.072160534 +0000 UTC m=+1613.769327837" Oct 31 03:22:10.524286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.522794 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:10.526513 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.526482 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:22:10.526761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.526733 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 03:22:10.565436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.565404 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:10.566955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.566928 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:22:10.567087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:10.566985 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 03:22:11.525409 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:11.525374 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:22:11.526030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:11.526001 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 03:22:11.565995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:11.565963 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:22:11.566183 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:11.566018 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 03:22:12.522275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:12.522020 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:22:12.522275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:12.522078 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 03:22:12.565734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:12.565699 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:22:12.566159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:12.565760 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 03:22:13.524670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:13.524634 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:22:13.524959 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:13.524928 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 03:22:13.566109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:13.566086 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:22:13.566540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:13.566519 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 03:22:14.527275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.525648 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:22:14.527275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.525707 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 03:22:14.567278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.566871 4854 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 03:22:14.567278 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.566929 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 03:22:14.819402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.755335 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn"] Oct 31 03:22:14.819402 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.813443 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" containerID="cri-o://8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0" gracePeriod=30 Oct 31 03:22:14.962320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.934715 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc"] Oct 31 03:22:14.962819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:14.962781 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" containerID="cri-o://cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a" gracePeriod=30 Oct 31 03:22:19.631708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:19.631676 4854 scope.go:122] "RemoveContainer" containerID="188d359394a320293e0deb007e4d7372627e98a307716f9aa893b5b12dca5a2a" Oct 31 03:22:19.642083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:19.642060 4854 scope.go:122] "RemoveContainer" containerID="fb259279580f2f9e0a24a084755b728bd857d9ae149bc2780d7c877684ec3792" Oct 31 03:22:20.138800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:20.138770 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94"] Oct 31 03:22:21.920654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.920629 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:21.938784 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.938762 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:21.994541 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994508 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.994705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994572 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.994705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994605 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.994705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994652 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.994705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994703 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.994936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994766 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.994936 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994869 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.995108 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.994950 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:21.995108 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995061 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.995221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995135 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.995325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995278 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.995398 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995332 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.995462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995413 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.995569 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995538 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.995789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995768 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.995886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995825 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995949 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.995996 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996038 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996064 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996127 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996187 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data\") pod \"492d17b7-7dca-48ca-991e-62e32e839673\" (UID: \"492d17b7-7dca-48ca-991e-62e32e839673\") " Oct 31 03:22:21.996428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996216 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy\") pod \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\" (UID: \"03a112cb-cb85-47b4-b21c-dcef4e92ac61\") " Oct 31 03:22:21.996871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996443 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.997004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.996980 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.997078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.997011 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.997078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.997025 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.997078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.997039 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.997078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.997052 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.997078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.997065 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:21.998693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.998666 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:21.999071 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.998924 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.999308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.999163 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.999308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.999206 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:21.999308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.999296 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:22:21.999517 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.999409 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:22:21.999517 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:21.999470 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.000508 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.000469 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:22.000818 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.000795 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "kube-api-access-bd46x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:22.000910 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.000884 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy" pod "03a112cb-cb85-47b4-b21c-dcef4e92ac61" (UID: "03a112cb-cb85-47b4-b21c-dcef4e92ac61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.001221 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.001198 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "kube-api-access-kzm2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:22.001630 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.001611 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token" pod "492d17b7-7dca-48ca-991e-62e32e839673" (UID: "492d17b7-7dca-48ca-991e-62e32e839673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:22.098515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098491 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098515 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098514 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzm2t\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-kube-api-access-kzm2t\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098524 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492d17b7-7dca-48ca-991e-62e32e839673-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098534 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098542 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098551 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492d17b7-7dca-48ca-991e-62e32e839673-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098559 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098566 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a112cb-cb85-47b4-b21c-dcef4e92ac61-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098575 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492d17b7-7dca-48ca-991e-62e32e839673-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098585 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd46x\" (UniqueName: \"kubernetes.io/projected/03a112cb-cb85-47b4-b21c-dcef4e92ac61-kube-api-access-bd46x\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098593 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.098654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.098600 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492d17b7-7dca-48ca-991e-62e32e839673-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.150481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.150407 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz"] Oct 31 03:22:22.150704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.150675 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" podUID="0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" containerName="maas-api" containerID="cri-o://f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29" gracePeriod=30 Oct 31 03:22:22.263011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.262974 4854 generic.go:356] "Generic (PLEG): container finished" podID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerID="8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0" exitCode=0 Oct 31 03:22:22.263410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.263385 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" event={"ID":"03a112cb-cb85-47b4-b21c-dcef4e92ac61","Type":"ContainerDied","Data":"8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0"} Oct 31 03:22:22.263634 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.263618 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" event={"ID":"03a112cb-cb85-47b4-b21c-dcef4e92ac61","Type":"ContainerDied","Data":"c21ada3bbe36588cf9fa0ac1efd9339c8f9f5869a68f4747667bd0e358b586a2"} Oct 31 03:22:22.263761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.263748 4854 scope.go:122] "RemoveContainer" containerID="8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0" Oct 31 03:22:22.264073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.264056 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn" Oct 31 03:22:22.266305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.266062 4854 generic.go:356] "Generic (PLEG): container finished" podID="492d17b7-7dca-48ca-991e-62e32e839673" containerID="cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a" exitCode=0 Oct 31 03:22:22.266305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.266190 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" event={"ID":"492d17b7-7dca-48ca-991e-62e32e839673","Type":"ContainerDied","Data":"cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a"} Oct 31 03:22:22.266305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.266228 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" event={"ID":"492d17b7-7dca-48ca-991e-62e32e839673","Type":"ContainerDied","Data":"8483cfeb172b35afbb1c1387fbac2e6080a02d4196bb331c7f47e2155dacd2d4"} Oct 31 03:22:22.266472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.266413 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc" Oct 31 03:22:22.278325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.278303 4854 scope.go:122] "RemoveContainer" containerID="8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0" Oct 31 03:22:22.278626 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:22.278607 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0\": container with ID starting with 8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0 not found: ID does not exist" containerID="8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0" Oct 31 03:22:22.278693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.278633 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0"} err="failed to get container status \"8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0\": rpc error: code = NotFound desc = could not find container \"8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0\": container with ID starting with 8b14e7853dd7d9001bd9fc6096ccfc0ce4388da99bbe64285157e4693dae51b0 not found: ID does not exist" Oct 31 03:22:22.278693 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.278648 4854 scope.go:122] "RemoveContainer" containerID="cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a" Oct 31 03:22:22.289628 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.289605 4854 scope.go:122] "RemoveContainer" containerID="cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a" Oct 31 03:22:22.289975 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:22.289951 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a\": container with ID starting with cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a not found: ID does not exist" containerID="cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a" Oct 31 03:22:22.290067 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.289988 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a"} err="failed to get container status \"cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a\": rpc error: code = NotFound desc = could not find container \"cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a\": container with ID starting with cfb85af845f8f0ecae24ddaecf2873d571ca2a3c609988ee2973d353af7e683a not found: ID does not exist" Oct 31 03:22:22.295337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.295311 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc"] Oct 31 03:22:22.299636 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.299612 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-lqccc"] Oct 31 03:22:22.313293 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.311917 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn"] Oct 31 03:22:22.316707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.316687 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-88jrn"] Oct 31 03:22:22.629068 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.629028 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" path="/var/lib/kubelet/pods/03a112cb-cb85-47b4-b21c-dcef4e92ac61/volumes" Oct 31 03:22:22.630211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.630097 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492d17b7-7dca-48ca-991e-62e32e839673" path="/var/lib/kubelet/pods/492d17b7-7dca-48ca-991e-62e32e839673/volumes" Oct 31 03:22:22.775283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.769052 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:22:22.868395 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.868366 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs\") pod \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " Oct 31 03:22:22.868533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.868492 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca\") pod \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " Oct 31 03:22:22.868594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.868556 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls\") pod \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " Oct 31 03:22:22.868631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.868615 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j\" (UniqueName: \"kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j\") pod \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\" (UID: \"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d\") " Oct 31 03:22:22.868877 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.868847 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca" pod "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" (UID: "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:22.869085 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.869066 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.870752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.870728 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs" pod "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" (UID: "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:22.870841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.870819 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j" pod "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" (UID: "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d"). InnerVolumeSpecName "kube-api-access-sxx4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:22.870879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.870865 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls" pod "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" (UID: "0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:22.970743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.970664 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.970743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.970693 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.970743 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:22.970703 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxx4j\" (UniqueName: \"kubernetes.io/projected/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d-kube-api-access-sxx4j\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:23.275290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.275225 4854 generic.go:356] "Generic (PLEG): container finished" podID="0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" containerID="f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29" exitCode=0 Oct 31 03:22:23.275501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.275310 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" Oct 31 03:22:23.275501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.275344 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" event={"ID":"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d","Type":"ContainerDied","Data":"f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29"} Oct 31 03:22:23.275501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.275385 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz" event={"ID":"0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d","Type":"ContainerDied","Data":"16374ffb66008fd3463ce603ee8c6f7c3d7c0202fc9e802b957d065fb03dcf97"} Oct 31 03:22:23.275501 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.275412 4854 scope.go:122] "RemoveContainer" containerID="f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29" Oct 31 03:22:23.287963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.287945 4854 scope.go:122] "RemoveContainer" containerID="f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29" Oct 31 03:22:23.288318 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:23.288295 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29\": container with ID starting with f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29 not found: ID does not exist" containerID="f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29" Oct 31 03:22:23.288426 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.288326 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29"} err="failed to get container status \"f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29\": rpc error: code = NotFound desc = could not find container \"f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29\": container with ID starting with f8fda9a44721e37c1123f88b639977f61eed7967674952db6441b6cd2e9fdf29 not found: ID does not exist" Oct 31 03:22:23.299570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.299540 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz"] Oct 31 03:22:23.302919 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:23.302898 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bbbc81-645d865f74-q9fhz"] Oct 31 03:22:24.457950 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:24.457912 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" path="/var/lib/kubelet/pods/0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d/volumes" Oct 31 03:22:25.582073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:25.582044 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf"] Oct 31 03:22:31.297562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:31.297507 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" podUID="53edaf58-a577-4597-8c91-104af425b533" containerName="payload-processing" containerID="cri-o://720efe4ea433af077d2bfdcb86e208009fe5e12a9dbaf5128bc32bc75664c237" gracePeriod=25 Oct 31 03:22:33.374759 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:33.374722 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" podUID="53edaf58-a577-4597-8c91-104af425b533" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:22:35.161795 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.161727 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" containerID="cri-o://c70c5dca47babe5f916b0fb1510450cca7ac055be845466a4be5fdefc7724f29" gracePeriod=45 Oct 31 03:22:35.372966 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.372928 4854 generic.go:356] "Generic (PLEG): container finished" podID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerID="c70c5dca47babe5f916b0fb1510450cca7ac055be845466a4be5fdefc7724f29" exitCode=0 Oct 31 03:22:35.373341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.373318 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" event={"ID":"42347c06-7df7-4f5c-acd3-9127f8f42302","Type":"ContainerDied","Data":"c70c5dca47babe5f916b0fb1510450cca7ac055be845466a4be5fdefc7724f29"} Oct 31 03:22:35.457647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.457618 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:22:35.525161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525126 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home\") pod \"42347c06-7df7-4f5c-acd3-9127f8f42302\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " Oct 31 03:22:35.525371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525214 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs\" (UniqueName: \"kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs\") pod \"42347c06-7df7-4f5c-acd3-9127f8f42302\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " Oct 31 03:22:35.525371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525283 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache\") pod \"42347c06-7df7-4f5c-acd3-9127f8f42302\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " Oct 31 03:22:35.525371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525326 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm\") pod \"42347c06-7df7-4f5c-acd3-9127f8f42302\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " Oct 31 03:22:35.525371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525351 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb\" (UniqueName: \"kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb\") pod \"42347c06-7df7-4f5c-acd3-9127f8f42302\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " Oct 31 03:22:35.525540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525432 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home" pod "42347c06-7df7-4f5c-acd3-9127f8f42302" (UID: "42347c06-7df7-4f5c-acd3-9127f8f42302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.525540 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525471 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir\") pod \"42347c06-7df7-4f5c-acd3-9127f8f42302\" (UID: \"42347c06-7df7-4f5c-acd3-9127f8f42302\") " Oct 31 03:22:35.525646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.525567 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache" pod "42347c06-7df7-4f5c-acd3-9127f8f42302" (UID: "42347c06-7df7-4f5c-acd3-9127f8f42302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.526080 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.526050 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir" pod "42347c06-7df7-4f5c-acd3-9127f8f42302" (UID: "42347c06-7df7-4f5c-acd3-9127f8f42302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.526276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.526241 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.526365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.526281 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.526365 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.526296 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.528530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.528503 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs" pod "42347c06-7df7-4f5c-acd3-9127f8f42302" (UID: "42347c06-7df7-4f5c-acd3-9127f8f42302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:35.528530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.528503 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm" pod "42347c06-7df7-4f5c-acd3-9127f8f42302" (UID: "42347c06-7df7-4f5c-acd3-9127f8f42302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.528657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.528570 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb" pod "42347c06-7df7-4f5c-acd3-9127f8f42302" (UID: "42347c06-7df7-4f5c-acd3-9127f8f42302"). InnerVolumeSpecName "kube-api-access-7zgfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:35.628185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.628154 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42347c06-7df7-4f5c-acd3-9127f8f42302-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.628185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.628181 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42347c06-7df7-4f5c-acd3-9127f8f42302-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.628185 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:35.628190 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zgfb\" (UniqueName: \"kubernetes.io/projected/42347c06-7df7-4f5c-acd3-9127f8f42302-kube-api-access-7zgfb\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:36.383775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:36.383735 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" Oct 31 03:22:36.383775 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:36.383745 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94" event={"ID":"42347c06-7df7-4f5c-acd3-9127f8f42302","Type":"ContainerDied","Data":"97825ec1a092366c200174a0a2d7fb9f5ae2d5f9900225277c21d1060b32e1e0"} Oct 31 03:22:36.384308 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:36.383798 4854 scope.go:122] "RemoveContainer" containerID="c70c5dca47babe5f916b0fb1510450cca7ac055be845466a4be5fdefc7724f29" Oct 31 03:22:36.405536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:36.405512 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94"] Oct 31 03:22:36.410777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:36.410754 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/premium-w6-7d8daf-kserve-66d8bc4788-k7g94"] Oct 31 03:22:36.457961 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:36.457935 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" path="/var/lib/kubelet/pods/42347c06-7df7-4f5c-acd3-9127f8f42302/volumes" Oct 31 03:22:39.251292 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:39.251239 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:22:43.397039 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:43.396578 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" podUID="53edaf58-a577-4597-8c91-104af425b533" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:22:50.443984 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:50.443948 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9"] Oct 31 03:22:50.444490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:50.444210 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" podUID="521608fc-f682-4cf0-96c2-f529ae646f78" containerName="maas-api" containerID="cri-o://e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b" gracePeriod=30 Oct 31 03:22:50.687960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:50.687583 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" podUID="521608fc-f682-4cf0-96c2-f529ae646f78" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.101:8443/health\": dial tcp 10.132.0.101:8443: connect: connection refused" Oct 31 03:22:51.009193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.009164 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:22:51.166732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.166642 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc\" (UniqueName: \"kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc\") pod \"521608fc-f682-4cf0-96c2-f529ae646f78\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " Oct 31 03:22:51.166732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.166722 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca\") pod \"521608fc-f682-4cf0-96c2-f529ae646f78\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " Oct 31 03:22:51.166980 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.166894 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls\") pod \"521608fc-f682-4cf0-96c2-f529ae646f78\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " Oct 31 03:22:51.167059 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.167040 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs\") pod \"521608fc-f682-4cf0-96c2-f529ae646f78\" (UID: \"521608fc-f682-4cf0-96c2-f529ae646f78\") " Oct 31 03:22:51.168650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.168618 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca" pod "521608fc-f682-4cf0-96c2-f529ae646f78" (UID: "521608fc-f682-4cf0-96c2-f529ae646f78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:51.171661 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.171632 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls" pod "521608fc-f682-4cf0-96c2-f529ae646f78" (UID: "521608fc-f682-4cf0-96c2-f529ae646f78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:51.171785 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.171654 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs" pod "521608fc-f682-4cf0-96c2-f529ae646f78" (UID: "521608fc-f682-4cf0-96c2-f529ae646f78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:51.171945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.171914 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc" pod "521608fc-f682-4cf0-96c2-f529ae646f78" (UID: "521608fc-f682-4cf0-96c2-f529ae646f78"). InnerVolumeSpecName "kube-api-access-znfkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:51.269019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.268992 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znfkc\" (UniqueName: \"kubernetes.io/projected/521608fc-f682-4cf0-96c2-f529ae646f78-kube-api-access-znfkc\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.269019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.269016 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521608fc-f682-4cf0-96c2-f529ae646f78-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.269149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.269028 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.269149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.269037 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/521608fc-f682-4cf0-96c2-f529ae646f78-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.495886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.495680 4854 generic.go:356] "Generic (PLEG): container finished" podID="521608fc-f682-4cf0-96c2-f529ae646f78" containerID="e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b" exitCode=0 Oct 31 03:22:51.495886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.495761 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" Oct 31 03:22:51.495886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.495765 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" event={"ID":"521608fc-f682-4cf0-96c2-f529ae646f78","Type":"ContainerDied","Data":"e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b"} Oct 31 03:22:51.495886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.495841 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9" event={"ID":"521608fc-f682-4cf0-96c2-f529ae646f78","Type":"ContainerDied","Data":"123e47f66bfffbf51376d521753c1a294ad614850eba0300fdb9793bc802bee8"} Oct 31 03:22:51.495886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.495875 4854 scope.go:122] "RemoveContainer" containerID="e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b" Oct 31 03:22:51.502155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.500113 4854 generic.go:356] "Generic (PLEG): container finished" podID="53edaf58-a577-4597-8c91-104af425b533" containerID="720efe4ea433af077d2bfdcb86e208009fe5e12a9dbaf5128bc32bc75664c237" exitCode=0 Oct 31 03:22:51.502155 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.500219 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" event={"ID":"53edaf58-a577-4597-8c91-104af425b533","Type":"ContainerDied","Data":"720efe4ea433af077d2bfdcb86e208009fe5e12a9dbaf5128bc32bc75664c237"} Oct 31 03:22:51.510375 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.510361 4854 scope.go:122] "RemoveContainer" containerID="e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b" Oct 31 03:22:51.510653 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:22:51.510635 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b\": container with ID starting with e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b not found: ID does not exist" containerID="e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b" Oct 31 03:22:51.510732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.510664 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b"} err="failed to get container status \"e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b\": rpc error: code = NotFound desc = could not find container \"e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b\": container with ID starting with e4005cadeb3542c6a6516a8374d767f9da35d3be459469c10c1e7941c015816b not found: ID does not exist" Oct 31 03:22:51.526292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.524976 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9"] Oct 31 03:22:51.529939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.529920 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7692c4-545f8d494c-7hjj9"] Oct 31 03:22:51.593791 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.593771 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:22:51.674052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.674020 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn\" (UniqueName: \"kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn\") pod \"53edaf58-a577-4597-8c91-104af425b533\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " Oct 31 03:22:51.674197 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.674123 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume\") pod \"53edaf58-a577-4597-8c91-104af425b533\" (UID: \"53edaf58-a577-4597-8c91-104af425b533\") " Oct 31 03:22:51.674506 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.674486 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume" pod "53edaf58-a577-4597-8c91-104af425b533" (UID: "53edaf58-a577-4597-8c91-104af425b533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:51.674736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.674722 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53edaf58-a577-4597-8c91-104af425b533-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.676461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.676439 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn" pod "53edaf58-a577-4597-8c91-104af425b533" (UID: "53edaf58-a577-4597-8c91-104af425b533"). InnerVolumeSpecName "kube-api-access-n27dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:51.775741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:51.775710 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27dn\" (UniqueName: \"kubernetes.io/projected/53edaf58-a577-4597-8c91-104af425b533-kube-api-access-n27dn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:52.457726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:52.457694 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521608fc-f682-4cf0-96c2-f529ae646f78" path="/var/lib/kubelet/pods/521608fc-f682-4cf0-96c2-f529ae646f78/volumes" Oct 31 03:22:52.508476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:52.508447 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" Oct 31 03:22:52.508834 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:52.508478 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf" event={"ID":"53edaf58-a577-4597-8c91-104af425b533","Type":"ContainerDied","Data":"09829da88c70e0956f34a13faba5f29f6d3fe23b70f867d196a5fe8ca43f8aa3"} Oct 31 03:22:52.508834 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:52.508532 4854 scope.go:122] "RemoveContainer" containerID="720efe4ea433af077d2bfdcb86e208009fe5e12a9dbaf5128bc32bc75664c237" Oct 31 03:22:52.524551 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:52.524526 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf"] Oct 31 03:22:52.529217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:52.529181 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d8daf-6559fc755-4zlrf"] Oct 31 03:22:54.458054 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:22:54.458017 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53edaf58-a577-4597-8c91-104af425b533" path="/var/lib/kubelet/pods/53edaf58-a577-4597-8c91-104af425b533/volumes" Oct 31 03:23:01.282371 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:01.282293 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b"] Oct 31 03:23:01.282786 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:01.282527 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" containerID="cri-o://fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab" gracePeriod=30 Oct 31 03:23:04.483802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:04.483769 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf"] Oct 31 03:23:04.484332 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:04.483992 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" containerID="cri-o://6747f426c24665ff2025e9b26a90fd1769cef739167284d6c398fd954f403e0a" gracePeriod=30 Oct 31 03:23:05.095226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:05.095191 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:05.095448 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:05.095239 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:06.573926 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.573901 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:23:06.696381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696354 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696419 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696464 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696521 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696570 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696549 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696581 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696647 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696815 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696794 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.696981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696909 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket\") pod \"fbc26c32-e947-4d59-9238-abd4cf8d261c\" (UID: \"fbc26c32-e947-4d59-9238-abd4cf8d261c\") " Oct 31 03:23:06.697040 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.696996 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:06.697166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.697131 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:06.697166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.697153 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:06.697330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.697310 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:06.697389 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.697360 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:06.697999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.697973 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc26c32-e947-4d59-9238-abd4cf8d261c-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.697999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.698001 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.698243 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.698016 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.698243 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.698030 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.698243 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.698045 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.699158 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.699131 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:06.699621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.699591 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:06.699749 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.699730 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:06.703415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.703392 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng" pod "fbc26c32-e947-4d59-9238-abd4cf8d261c" (UID: "fbc26c32-e947-4d59-9238-abd4cf8d261c"). InnerVolumeSpecName "kube-api-access-sxsng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:06.723777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.723751 4854 generic.go:356] "Generic (PLEG): container finished" podID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerID="fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab" exitCode=0 Oct 31 03:23:06.723889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.723803 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" Oct 31 03:23:06.723889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.723813 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" event={"ID":"fbc26c32-e947-4d59-9238-abd4cf8d261c","Type":"ContainerDied","Data":"fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab"} Oct 31 03:23:06.723889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.723856 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b" event={"ID":"fbc26c32-e947-4d59-9238-abd4cf8d261c","Type":"ContainerDied","Data":"94e0f69b8014388fd874ed057143927c65d75b6d39d0e2ea084e0a082d3fba30"} Oct 31 03:23:06.723889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.723877 4854 scope.go:122] "RemoveContainer" containerID="fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab" Oct 31 03:23:06.737805 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.737777 4854 scope.go:122] "RemoveContainer" containerID="059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964" Oct 31 03:23:06.747590 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.747574 4854 scope.go:122] "RemoveContainer" containerID="fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab" Oct 31 03:23:06.747861 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:06.747838 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab\": container with ID starting with fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab not found: ID does not exist" containerID="fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab" Oct 31 03:23:06.747925 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.747873 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab"} err="failed to get container status \"fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab\": rpc error: code = NotFound desc = could not find container \"fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab\": container with ID starting with fdff0753804ccd44603d45dff28b0394bedc57ba15ada01cc38fe65bf46f9eab not found: ID does not exist" Oct 31 03:23:06.747925 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.747890 4854 scope.go:122] "RemoveContainer" containerID="059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964" Oct 31 03:23:06.748222 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:06.748194 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964\": container with ID starting with 059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964 not found: ID does not exist" containerID="059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964" Oct 31 03:23:06.748373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.748233 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964"} err="failed to get container status \"059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964\": rpc error: code = NotFound desc = could not find container \"059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964\": container with ID starting with 059a725b6b2a71d07f46eacadb6532a4e3ee7f3945f1076018f18ae6cae32964 not found: ID does not exist" Oct 31 03:23:06.750733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.750714 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b"] Oct 31 03:23:06.755577 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.755554 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7692c4-openshift-default-7dc7d87fbf-7bv4b"] Oct 31 03:23:06.799347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.799286 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.799347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.799312 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.799347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.799327 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxsng\" (UniqueName: \"kubernetes.io/projected/fbc26c32-e947-4d59-9238-abd4cf8d261c-kube-api-access-sxsng\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:06.799347 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:06.799343 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc26c32-e947-4d59-9238-abd4cf8d261c-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:07.084973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:07.084887 4854 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:07.084973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:07.084954 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:08.457186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:08.457149 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" path="/var/lib/kubelet/pods/fbc26c32-e947-4d59-9238-abd4cf8d261c/volumes" Oct 31 03:23:09.251825 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:09.251791 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:23:09.773826 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.772352 4854 generic.go:356] "Generic (PLEG): container finished" podID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerID="6747f426c24665ff2025e9b26a90fd1769cef739167284d6c398fd954f403e0a" exitCode=0 Oct 31 03:23:09.773826 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.772564 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" event={"ID":"a463dafe-3ebd-4c43-9376-934853ff4b36","Type":"ContainerDied","Data":"6747f426c24665ff2025e9b26a90fd1769cef739167284d6c398fd954f403e0a"} Oct 31 03:23:09.840523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.840501 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:23:09.964435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964401 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964441 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964540 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964580 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964600 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964635 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964682 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.964840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964689 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.964840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964768 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.965066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.964866 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert\") pod \"a463dafe-3ebd-4c43-9376-934853ff4b36\" (UID: \"a463dafe-3ebd-4c43-9376-934853ff4b36\") " Oct 31 03:23:09.965127 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.965061 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.965235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.965209 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.965491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.965468 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:09.966298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.966022 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.966298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.966051 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a463dafe-3ebd-4c43-9376-934853ff4b36-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.966298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.966066 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.966298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.966080 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.966298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.966269 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.967230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.967206 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:09.967355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.967297 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.967533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.967508 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "kube-api-access-cb4f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.967942 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:09.967921 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token" pod "a463dafe-3ebd-4c43-9376-934853ff4b36" (UID: "a463dafe-3ebd-4c43-9376-934853ff4b36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:10.067303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.067204 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4f4\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-kube-api-access-cb4f4\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.067303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.067239 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.067303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.067269 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.067303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.067281 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.067303 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.067294 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a463dafe-3ebd-4c43-9376-934853ff4b36-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.779929 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.779894 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" Oct 31 03:23:10.780339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.779943 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf" event={"ID":"a463dafe-3ebd-4c43-9376-934853ff4b36","Type":"ContainerDied","Data":"636dfa5bcf5c508168e264b74cf4b4cf1e4e2b6eae33b9b9bc1e14fe3dea644b"} Oct 31 03:23:10.780339 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.780004 4854 scope.go:122] "RemoveContainer" containerID="6747f426c24665ff2025e9b26a90fd1769cef739167284d6c398fd954f403e0a" Oct 31 03:23:10.797888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.797865 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf"] Oct 31 03:23:10.806235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:10.806208 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-cxfgf"] Oct 31 03:23:12.034685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:12.034653 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v"] Oct 31 03:23:12.458476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:12.458392 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" path="/var/lib/kubelet/pods/a463dafe-3ebd-4c43-9376-934853ff4b36/volumes" Oct 31 03:23:14.711179 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:14.711143 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj"] Oct 31 03:23:14.711737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:14.711431 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" podUID="37b2af3c-0bd3-4866-b0f0-69572d366e40" containerName="maas-api" containerID="cri-o://4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f" gracePeriod=30 Oct 31 03:23:15.293129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.290117 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:23:15.379118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.379088 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls\") pod \"37b2af3c-0bd3-4866-b0f0-69572d366e40\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " Oct 31 03:23:15.379359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.379279 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs\") pod \"37b2af3c-0bd3-4866-b0f0-69572d366e40\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " Oct 31 03:23:15.379359 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.379347 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca\") pod \"37b2af3c-0bd3-4866-b0f0-69572d366e40\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " Oct 31 03:23:15.379503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.379371 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd\") pod \"37b2af3c-0bd3-4866-b0f0-69572d366e40\" (UID: \"37b2af3c-0bd3-4866-b0f0-69572d366e40\") " Oct 31 03:23:15.379716 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.379683 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca" pod "37b2af3c-0bd3-4866-b0f0-69572d366e40" (UID: "37b2af3c-0bd3-4866-b0f0-69572d366e40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:15.379950 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.379874 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b2af3c-0bd3-4866-b0f0-69572d366e40-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:15.381505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.381484 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls" pod "37b2af3c-0bd3-4866-b0f0-69572d366e40" (UID: "37b2af3c-0bd3-4866-b0f0-69572d366e40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:15.381600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.381575 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs" pod "37b2af3c-0bd3-4866-b0f0-69572d366e40" (UID: "37b2af3c-0bd3-4866-b0f0-69572d366e40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:15.381656 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.381637 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd" pod "37b2af3c-0bd3-4866-b0f0-69572d366e40" (UID: "37b2af3c-0bd3-4866-b0f0-69572d366e40"). InnerVolumeSpecName "kube-api-access-2nbdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:15.480757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.480677 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:15.480757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.480703 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b2af3c-0bd3-4866-b0f0-69572d366e40-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:15.480757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.480713 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/37b2af3c-0bd3-4866-b0f0-69572d366e40-kube-api-access-2nbdd\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:15.820737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.820701 4854 generic.go:356] "Generic (PLEG): container finished" podID="37b2af3c-0bd3-4866-b0f0-69572d366e40" containerID="4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f" exitCode=0 Oct 31 03:23:15.821220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.820748 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" Oct 31 03:23:15.821220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.820781 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" event={"ID":"37b2af3c-0bd3-4866-b0f0-69572d366e40","Type":"ContainerDied","Data":"4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f"} Oct 31 03:23:15.821220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.820846 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj" event={"ID":"37b2af3c-0bd3-4866-b0f0-69572d366e40","Type":"ContainerDied","Data":"bccbc1435aaa699db1a1d3184d3a706a1f06ceb6d487ec2575cd1070d52b6e76"} Oct 31 03:23:15.821220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.820876 4854 scope.go:122] "RemoveContainer" containerID="4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f" Oct 31 03:23:15.833342 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.833324 4854 scope.go:122] "RemoveContainer" containerID="4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f" Oct 31 03:23:15.833640 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:15.833619 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f\": container with ID starting with 4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f not found: ID does not exist" containerID="4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f" Oct 31 03:23:15.833702 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.833647 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f"} err="failed to get container status \"4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f\": rpc error: code = NotFound desc = could not find container \"4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f\": container with ID starting with 4891be47e62d322469a8659811ddd541c5fd76b9dc980378efdb12ab8f51b39f not found: ID does not exist" Oct 31 03:23:15.857310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.857286 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj"] Oct 31 03:23:15.861230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:15.861209 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3fc2d0-67688c9c59-dtwrj"] Oct 31 03:23:16.457023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:16.456993 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b2af3c-0bd3-4866-b0f0-69572d366e40" path="/var/lib/kubelet/pods/37b2af3c-0bd3-4866-b0f0-69572d366e40/volumes" Oct 31 03:23:17.034977 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.034938 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" podUID="5eddc677-2ef2-412a-8dda-f2e80581ad59" containerName="payload-processing" containerID="cri-o://1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd" gracePeriod=25 Oct 31 03:23:17.323159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.323134 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:23:17.452118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.451449 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp\" (UniqueName: \"kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp\") pod \"5eddc677-2ef2-412a-8dda-f2e80581ad59\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " Oct 31 03:23:17.452118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.451563 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume\") pod \"5eddc677-2ef2-412a-8dda-f2e80581ad59\" (UID: \"5eddc677-2ef2-412a-8dda-f2e80581ad59\") " Oct 31 03:23:17.452450 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.452422 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume" pod "5eddc677-2ef2-412a-8dda-f2e80581ad59" (UID: "5eddc677-2ef2-412a-8dda-f2e80581ad59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:17.454380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.454356 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp" pod "5eddc677-2ef2-412a-8dda-f2e80581ad59" (UID: "5eddc677-2ef2-412a-8dda-f2e80581ad59"). InnerVolumeSpecName "kube-api-access-4gtvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:17.553623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.553505 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtvp\" (UniqueName: \"kubernetes.io/projected/5eddc677-2ef2-412a-8dda-f2e80581ad59-kube-api-access-4gtvp\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:17.553623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.553541 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eddc677-2ef2-412a-8dda-f2e80581ad59-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:17.837483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.837398 4854 generic.go:356] "Generic (PLEG): container finished" podID="5eddc677-2ef2-412a-8dda-f2e80581ad59" containerID="1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd" exitCode=0 Oct 31 03:23:17.837483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.837463 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" event={"ID":"5eddc677-2ef2-412a-8dda-f2e80581ad59","Type":"ContainerDied","Data":"1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd"} Oct 31 03:23:17.837483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.837473 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" Oct 31 03:23:17.837726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.837547 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v" event={"ID":"5eddc677-2ef2-412a-8dda-f2e80581ad59","Type":"ContainerDied","Data":"64fbc576725cb0ff9540eca6415291f8c7b2b69d2b140db74010c55a93ae64bf"} Oct 31 03:23:17.837726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.837577 4854 scope.go:122] "RemoveContainer" containerID="1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd" Oct 31 03:23:17.847682 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.847661 4854 scope.go:122] "RemoveContainer" containerID="1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd" Oct 31 03:23:17.847924 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:17.847905 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd\": container with ID starting with 1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd not found: ID does not exist" containerID="1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd" Oct 31 03:23:17.847999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.847933 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd"} err="failed to get container status \"1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd\": rpc error: code = NotFound desc = could not find container \"1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd\": container with ID starting with 1ce4a5b7bd7d9584b669fed7dfd6ba1cef69d3d17aaca922d09b05f5322fe3fd not found: ID does not exist" Oct 31 03:23:17.859955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.859934 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v"] Oct 31 03:23:17.863681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:17.863659 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3fc2d0-8ffcccb75-fl75v"] Oct 31 03:23:18.457827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:18.457779 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eddc677-2ef2-412a-8dda-f2e80581ad59" path="/var/lib/kubelet/pods/5eddc677-2ef2-412a-8dda-f2e80581ad59/volumes" Oct 31 03:23:29.233839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:29.233797 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9"] Oct 31 03:23:29.240462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:29.239966 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc"] Oct 31 03:23:34.602327 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:34.602290 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw"] Oct 31 03:23:34.615825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:34.615795 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g"] Oct 31 03:23:39.251198 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:39.251162 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:23:39.603677 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:39.603590 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerName="payload-processing" containerID="cri-o://7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf" gracePeriod=25 Oct 31 03:23:39.617117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:39.617072 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerName="payload-pre-processing" containerID="cri-o://32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f" gracePeriod=25 Oct 31 03:23:43.374200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:43.374156 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:23:43.374713 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:43.374171 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:23:44.257847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.257784 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" containerID="cri-o://f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf" gracePeriod=45 Oct 31 03:23:44.263803 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.263771 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" containerID="cri-o://9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d" gracePeriod=45 Oct 31 03:23:44.566063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.566032 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:23:44.660667 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.660645 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:23:44.681597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.681566 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj\" (UniqueName: \"kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj\") pod \"b2ce5a45-f5ec-4a49-a765-440318032ee5\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " Oct 31 03:23:44.681734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.681685 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm\") pod \"b2ce5a45-f5ec-4a49-a765-440318032ee5\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " Oct 31 03:23:44.681779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.681733 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir\") pod \"b2ce5a45-f5ec-4a49-a765-440318032ee5\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " Oct 31 03:23:44.681843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.681820 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs\") pod \"b2ce5a45-f5ec-4a49-a765-440318032ee5\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " Oct 31 03:23:44.682662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.682598 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache\") pod \"b2ce5a45-f5ec-4a49-a765-440318032ee5\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " Oct 31 03:23:44.682830 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.682798 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home\") pod \"b2ce5a45-f5ec-4a49-a765-440318032ee5\" (UID: \"b2ce5a45-f5ec-4a49-a765-440318032ee5\") " Oct 31 03:23:44.683509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.683483 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache" pod "b2ce5a45-f5ec-4a49-a765-440318032ee5" (UID: "b2ce5a45-f5ec-4a49-a765-440318032ee5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.683650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.683630 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.683734 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.683655 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home" pod "b2ce5a45-f5ec-4a49-a765-440318032ee5" (UID: "b2ce5a45-f5ec-4a49-a765-440318032ee5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.685065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.685033 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj" pod "b2ce5a45-f5ec-4a49-a765-440318032ee5" (UID: "b2ce5a45-f5ec-4a49-a765-440318032ee5"). InnerVolumeSpecName "kube-api-access-jq8cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:44.685500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.685463 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir" pod "b2ce5a45-f5ec-4a49-a765-440318032ee5" (UID: "b2ce5a45-f5ec-4a49-a765-440318032ee5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.686439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.685733 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs" pod "b2ce5a45-f5ec-4a49-a765-440318032ee5" (UID: "b2ce5a45-f5ec-4a49-a765-440318032ee5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:44.687291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.686653 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm" pod "b2ce5a45-f5ec-4a49-a765-440318032ee5" (UID: "b2ce5a45-f5ec-4a49-a765-440318032ee5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.784967 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.784895 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir\") pod \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " Oct 31 03:23:44.785116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.784973 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache\") pod \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " Oct 31 03:23:44.785116 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785002 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs\") pod \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " Oct 31 03:23:44.785286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785140 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm\") pod \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " Oct 31 03:23:44.785286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785217 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache" pod "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" (UID: "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.785286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785233 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir" pod "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" (UID: "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.785286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785225 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8\" (UniqueName: \"kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8\") pod \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " Oct 31 03:23:44.785499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785316 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home\") pod \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\" (UID: \"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0\") " Oct 31 03:23:44.785605 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785579 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home" pod "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" (UID: "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.785937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785922 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.785937 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785936 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.786073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785945 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ce5a45-f5ec-4a49-a765-440318032ee5-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.786073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785953 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.786073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785961 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.786073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785968 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.786073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785978 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8cj\" (UniqueName: \"kubernetes.io/projected/b2ce5a45-f5ec-4a49-a765-440318032ee5-kube-api-access-jq8cj\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.786073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.785986 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ce5a45-f5ec-4a49-a765-440318032ee5-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.787376 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.787356 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8" pod "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" (UID: "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0"). InnerVolumeSpecName "kube-api-access-2l2z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:44.787508 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.787488 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs" pod "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" (UID: "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:44.787557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.787518 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm" pod "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" (UID: "e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.888044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.888013 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.888044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.888041 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.888044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:44.888052 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l2z8\" (UniqueName: \"kubernetes.io/projected/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0-kube-api-access-2l2z8\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:45.156881 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.156811 4854 generic.go:356] "Generic (PLEG): container finished" podID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerID="f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf" exitCode=0 Oct 31 03:23:45.156881 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.156852 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" Oct 31 03:23:45.157091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.156894 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" event={"ID":"b2ce5a45-f5ec-4a49-a765-440318032ee5","Type":"ContainerDied","Data":"f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf"} Oct 31 03:23:45.157091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.156934 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9" event={"ID":"b2ce5a45-f5ec-4a49-a765-440318032ee5","Type":"ContainerDied","Data":"91bba88a8300e7383ca5e2da3135fa8e4b81d35757ace71e6a4e430a1b3766ff"} Oct 31 03:23:45.157091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.156956 4854 scope.go:122] "RemoveContainer" containerID="f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf" Oct 31 03:23:45.158579 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.158555 4854 generic.go:356] "Generic (PLEG): container finished" podID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerID="9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d" exitCode=0 Oct 31 03:23:45.158711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.158597 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" Oct 31 03:23:45.158711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.158639 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" event={"ID":"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0","Type":"ContainerDied","Data":"9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d"} Oct 31 03:23:45.158711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.158678 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc" event={"ID":"e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0","Type":"ContainerDied","Data":"1064ecca76861f3cebbdd280d558955234233d71586ae90f4428d35ff32030a0"} Oct 31 03:23:45.167360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.167345 4854 scope.go:122] "RemoveContainer" containerID="f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf" Oct 31 03:23:45.167651 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:45.167634 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf\": container with ID starting with f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf not found: ID does not exist" containerID="f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf" Oct 31 03:23:45.167697 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.167658 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf"} err="failed to get container status \"f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf\": rpc error: code = NotFound desc = could not find container \"f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf\": container with ID starting with f862fddcfdbe5bc260f6be2fe91fb732af398c406d88b123c024abc3bd839ecf not found: ID does not exist" Oct 31 03:23:45.167697 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.167676 4854 scope.go:122] "RemoveContainer" containerID="9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d" Oct 31 03:23:45.177431 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.177413 4854 scope.go:122] "RemoveContainer" containerID="9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d" Oct 31 03:23:45.177703 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:23:45.177684 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d\": container with ID starting with 9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d not found: ID does not exist" containerID="9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d" Oct 31 03:23:45.177751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.177712 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d"} err="failed to get container status \"9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d\": rpc error: code = NotFound desc = could not find container \"9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d\": container with ID starting with 9658922440edc7e23e5dd1d8702aea9b2f626fed3786e416df6477cce7bcb42d not found: ID does not exist" Oct 31 03:23:45.184462 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.184439 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9"] Oct 31 03:23:45.187237 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.187214 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-2-w2-85cb59-kserve-6f9d5d48b4-6v4d9"] Oct 31 03:23:45.195862 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.195841 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc"] Oct 31 03:23:45.202338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:45.202318 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/distinct-w2-85cb59-kserve-66cccf77dc-vflkc"] Oct 31 03:23:46.457186 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:46.457152 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" path="/var/lib/kubelet/pods/b2ce5a45-f5ec-4a49-a765-440318032ee5/volumes" Oct 31 03:23:46.457614 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:46.457600 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" path="/var/lib/kubelet/pods/e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0/volumes" Oct 31 03:23:53.373959 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:53.373923 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:23:53.374467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:23:53.373952 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:00.090550 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.090526 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:24:00.120522 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.120496 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:24:00.133073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.133043 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn\" (UniqueName: \"kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn\") pod \"a847ac29-5362-45ac-9715-77a1c1a90e2b\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " Oct 31 03:24:00.133202 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.133179 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume\") pod \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " Oct 31 03:24:00.133285 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.133241 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume\") pod \"a847ac29-5362-45ac-9715-77a1c1a90e2b\" (UID: \"a847ac29-5362-45ac-9715-77a1c1a90e2b\") " Oct 31 03:24:00.133482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.133461 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4\" (UniqueName: \"kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4\") pod \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\" (UID: \"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf\") " Oct 31 03:24:00.133888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.133859 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume" pod "743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" (UID: "743857fb-f1c4-4cc7-a6c3-ba01a0e070cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:00.134008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.133911 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume" pod "a847ac29-5362-45ac-9715-77a1c1a90e2b" (UID: "a847ac29-5362-45ac-9715-77a1c1a90e2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:00.134291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.134229 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:00.134403 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.134298 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a847ac29-5362-45ac-9715-77a1c1a90e2b-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:00.135906 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.135885 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn" pod "a847ac29-5362-45ac-9715-77a1c1a90e2b" (UID: "a847ac29-5362-45ac-9715-77a1c1a90e2b"). InnerVolumeSpecName "kube-api-access-g4rjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:00.136556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.136521 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4" pod "743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" (UID: "743857fb-f1c4-4cc7-a6c3-ba01a0e070cf"). InnerVolumeSpecName "kube-api-access-b7jl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:00.236044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.236012 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jl4\" (UniqueName: \"kubernetes.io/projected/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf-kube-api-access-b7jl4\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:00.236044 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.236040 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rjn\" (UniqueName: \"kubernetes.io/projected/a847ac29-5362-45ac-9715-77a1c1a90e2b-kube-api-access-g4rjn\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:00.291504 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.291443 4854 generic.go:356] "Generic (PLEG): container finished" podID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerID="32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f" exitCode=0 Oct 31 03:24:00.291504 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.291489 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" Oct 31 03:24:00.291659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.291533 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" event={"ID":"a847ac29-5362-45ac-9715-77a1c1a90e2b","Type":"ContainerDied","Data":"32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f"} Oct 31 03:24:00.291659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.291593 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g" event={"ID":"a847ac29-5362-45ac-9715-77a1c1a90e2b","Type":"ContainerDied","Data":"3c38db24ad58913bfd1088381cbc94bc688d19321e2810fc3cf0406f8be5ac27"} Oct 31 03:24:00.291659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.291621 4854 scope.go:122] "RemoveContainer" containerID="32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f" Oct 31 03:24:00.293139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.293115 4854 generic.go:356] "Generic (PLEG): container finished" podID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerID="7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf" exitCode=0 Oct 31 03:24:00.293235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.293168 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" Oct 31 03:24:00.293235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.293174 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" event={"ID":"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf","Type":"ContainerDied","Data":"7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf"} Oct 31 03:24:00.293235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.293227 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw" event={"ID":"743857fb-f1c4-4cc7-a6c3-ba01a0e070cf","Type":"ContainerDied","Data":"83ae60f57cebd6dd2eeeb4e506c6a65c43f2765b9594cbb5755a44144fd54cd6"} Oct 31 03:24:00.302842 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.302821 4854 scope.go:122] "RemoveContainer" containerID="32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f" Oct 31 03:24:00.303194 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:00.303170 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f\": container with ID starting with 32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f not found: ID does not exist" containerID="32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f" Oct 31 03:24:00.303333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.303200 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f"} err="failed to get container status \"32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f\": rpc error: code = NotFound desc = could not find container \"32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f\": container with ID starting with 32e20c9e4c7c9fefe89d1a91159fba7b48b230fb83b12bfb6007bf3f2e96ba9f not found: ID does not exist" Oct 31 03:24:00.303333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.303218 4854 scope.go:122] "RemoveContainer" containerID="7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf" Oct 31 03:24:00.312572 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.312536 4854 scope.go:122] "RemoveContainer" containerID="7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf" Oct 31 03:24:00.312911 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:00.312881 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf\": container with ID starting with 7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf not found: ID does not exist" containerID="7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf" Oct 31 03:24:00.313112 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.312920 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf"} err="failed to get container status \"7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf\": rpc error: code = NotFound desc = could not find container \"7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf\": container with ID starting with 7382f2ddf17a12c674df6e97e9ad122afb47ae617cc235a811cfbc4677f746cf not found: ID does not exist" Oct 31 03:24:00.315681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.315659 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g"] Oct 31 03:24:00.317772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.317754 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-85cb59-5d5fd7bcb7-d6h8g"] Oct 31 03:24:00.325519 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.325498 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw"] Oct 31 03:24:00.327727 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.327709 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-85cb59-587d5dfc5f-bdhqw"] Oct 31 03:24:00.457338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.457311 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" path="/var/lib/kubelet/pods/743857fb-f1c4-4cc7-a6c3-ba01a0e070cf/volumes" Oct 31 03:24:00.457711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:00.457697 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" path="/var/lib/kubelet/pods/a847ac29-5362-45ac-9715-77a1c1a90e2b/volumes" Oct 31 03:24:01.879129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:01.873330 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv"] Oct 31 03:24:01.879129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:01.873636 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" containerID="cri-o://203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef" gracePeriod=30 Oct 31 03:24:01.879129 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:01.875842 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z"] Oct 31 03:24:01.886493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:01.886438 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" containerID="cri-o://99c981e9504b87c3b84ca973013e0e0675a7801d0bd1c5cfa066247bb611578d" gracePeriod=30 Oct 31 03:24:02.806025 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:02.805988 4854 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:24:02.806184 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:02.806045 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:24:04.186195 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:04.186162 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8"] Oct 31 03:24:07.153804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.153774 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.235677 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.235764 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.235866 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.235932 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.235986 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.236081 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.236111 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.236145 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.244978 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.236207 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token\") pod \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\" (UID: \"67d33873-bc9b-4ccc-bee4-7e50adf8e730\") " Oct 31 03:24:07.280273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.269641 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.281396 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.281353 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.304269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.281733 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:07.358236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.352886 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.358236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.352924 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.358236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.352938 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.386286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.383371 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.386286 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.383716 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:07.409863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.409824 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.418284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.410761 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.433865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.432652 4854 generic.go:356] "Generic (PLEG): container finished" podID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerID="203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef" exitCode=0 Oct 31 03:24:07.433865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.432903 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" event={"ID":"67d33873-bc9b-4ccc-bee4-7e50adf8e730","Type":"ContainerDied","Data":"203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef"} Oct 31 03:24:07.433865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.432963 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" event={"ID":"67d33873-bc9b-4ccc-bee4-7e50adf8e730","Type":"ContainerDied","Data":"e9173327bcec2b262839f46327b7b94683582bb70246b15046baad5a0256f260"} Oct 31 03:24:07.433865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.432994 4854 scope.go:122] "RemoveContainer" containerID="203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef" Oct 31 03:24:07.433865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.433172 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv" Oct 31 03:24:07.440072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.439944 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" event={"ID":"9f386509-0bc2-4565-9b54-f4efa32fad2c","Type":"ContainerDied","Data":"99c981e9504b87c3b84ca973013e0e0675a7801d0bd1c5cfa066247bb611578d"} Oct 31 03:24:07.440433 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.439787 4854 generic.go:356] "Generic (PLEG): container finished" podID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerID="99c981e9504b87c3b84ca973013e0e0675a7801d0bd1c5cfa066247bb611578d" exitCode=0 Oct 31 03:24:07.441162 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.441134 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "kube-api-access-2kzk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.452574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.451998 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy" pod "67d33873-bc9b-4ccc-bee4-7e50adf8e730" (UID: "67d33873-bc9b-4ccc-bee4-7e50adf8e730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.455226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.454592 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.455226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.454636 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-kube-api-access-2kzk2\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.455226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.454672 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.455226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.454688 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d33873-bc9b-4ccc-bee4-7e50adf8e730-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.455226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.454702 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.455226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.454757 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d33873-bc9b-4ccc-bee4-7e50adf8e730-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.497198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.497178 4854 scope.go:122] "RemoveContainer" containerID="203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef" Oct 31 03:24:07.497955 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:07.497646 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef\": container with ID starting with 203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef not found: ID does not exist" containerID="203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef" Oct 31 03:24:07.497955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.497675 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef"} err="failed to get container status \"203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef\": rpc error: code = NotFound desc = could not find container \"203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef\": container with ID starting with 203b5712a29f18b53c47086777deb6a117e110c657ce81f6468d9ce870c819ef not found: ID does not exist" Oct 31 03:24:07.497955 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.497692 4854 scope.go:122] "RemoveContainer" containerID="4f30fa83e164edb8bdc65afcf80c1065e2fec2edfc22e9f66e85f625f0da2f8f" Oct 31 03:24:07.646357 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.646333 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:24:07.756006 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.755945 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv"] Oct 31 03:24:07.758799 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.758772 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.758899 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.758954 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759065 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759097 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759176 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759226 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759290 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759330 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert\") pod \"9f386509-0bc2-4565-9b54-f4efa32fad2c\" (UID: \"9f386509-0bc2-4565-9b54-f4efa32fad2c\") " Oct 31 03:24:07.759894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.759869 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.760292 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.760264 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.760690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.760364 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:07.761029 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.761003 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.761573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.761548 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.761944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.761922 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.762219 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.762199 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:07.762443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.762419 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.762668 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.762578 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx" pod "9f386509-0bc2-4565-9b54-f4efa32fad2c" (UID: "9f386509-0bc2-4565-9b54-f4efa32fad2c"). InnerVolumeSpecName "kube-api-access-57mpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.764045 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.764026 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-m7tvv"] Oct 31 03:24:07.861276 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861240 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861279 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861288 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861297 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861305 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861313 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f386509-0bc2-4565-9b54-f4efa32fad2c-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861323 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861331 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mpx\" (UniqueName: \"kubernetes.io/projected/9f386509-0bc2-4565-9b54-f4efa32fad2c-kube-api-access-57mpx\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.861355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:07.861340 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f386509-0bc2-4565-9b54-f4efa32fad2c-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:08.448536 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:08.448505 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" Oct 31 03:24:08.448992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:08.448500 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z" event={"ID":"9f386509-0bc2-4565-9b54-f4efa32fad2c","Type":"ContainerDied","Data":"d03679ff86fa2d10f0b5f417473592d8c870781d1d93ba0c4bfefccbca66d079"} Oct 31 03:24:08.448992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:08.448722 4854 scope.go:122] "RemoveContainer" containerID="99c981e9504b87c3b84ca973013e0e0675a7801d0bd1c5cfa066247bb611578d" Oct 31 03:24:08.458797 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:08.458771 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" path="/var/lib/kubelet/pods/67d33873-bc9b-4ccc-bee4-7e50adf8e730/volumes" Oct 31 03:24:08.473962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:08.473938 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z"] Oct 31 03:24:08.477964 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:08.477944 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-xcg7z"] Oct 31 03:24:09.186485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:09.186446 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerName="payload-pre-processing" containerID="cri-o://455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91" gracePeriod=25 Oct 31 03:24:09.250387 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:09.250355 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:24:10.457401 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:10.457371 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" path="/var/lib/kubelet/pods/9f386509-0bc2-4565-9b54-f4efa32fad2c/volumes" Oct 31 03:24:15.730321 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:15.730286 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:15.805131 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:15.805091 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf"] Oct 31 03:24:15.809464 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:15.809346 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" containerID="cri-o://29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d" gracePeriod=30 Oct 31 03:24:21.098759 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.098732 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:24:21.111750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.111718 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.111889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.111761 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.111960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.111942 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.111992 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112082 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112040 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112082 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112140 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112112 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112171 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:21.112271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112214 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112334 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket\") pod \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\" (UID: \"9661d7af-7aba-4d48-bf9a-1f97f804b29a\") " Oct 31 03:24:21.112573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112552 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:21.112715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112592 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:21.112807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112700 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:21.112807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.112727 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:21.113087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.113069 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.113126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.113094 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.113126 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.113110 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.113193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.113126 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.113193 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.113140 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.114190 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.114162 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "kube-api-access-sr6g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:21.114563 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.114535 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:21.114653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.114584 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:21.114703 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.114665 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token" pod "9661d7af-7aba-4d48-bf9a-1f97f804b29a" (UID: "9661d7af-7aba-4d48-bf9a-1f97f804b29a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:21.214385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.214353 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.214385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.214382 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6g7\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-kube-api-access-sr6g7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.214385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.214393 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.214613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.214402 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9661d7af-7aba-4d48-bf9a-1f97f804b29a-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:21.562781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.562746 4854 generic.go:356] "Generic (PLEG): container finished" podID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerID="29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d" exitCode=0 Oct 31 03:24:21.562931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.562797 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" Oct 31 03:24:21.562931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.562824 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" event={"ID":"9661d7af-7aba-4d48-bf9a-1f97f804b29a","Type":"ContainerDied","Data":"29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d"} Oct 31 03:24:21.562931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.562883 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf" event={"ID":"9661d7af-7aba-4d48-bf9a-1f97f804b29a","Type":"ContainerDied","Data":"792c63e1b9afc19c92a128ca6f9766b9c6ac9cf6c509a69aa4cea2f8137292c9"} Oct 31 03:24:21.562931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.562909 4854 scope.go:122] "RemoveContainer" containerID="29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d" Oct 31 03:24:21.573603 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.573585 4854 scope.go:122] "RemoveContainer" containerID="714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e" Oct 31 03:24:21.583505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.583473 4854 scope.go:122] "RemoveContainer" containerID="29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d" Oct 31 03:24:21.583776 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:21.583755 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d\": container with ID starting with 29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d not found: ID does not exist" containerID="29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d" Oct 31 03:24:21.583849 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.583788 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d"} err="failed to get container status \"29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d\": rpc error: code = NotFound desc = could not find container \"29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d\": container with ID starting with 29575d5e12ed13fcd48b531193d7bcaef69247c7f6f44a769d3c5a3dfbeca55d not found: ID does not exist" Oct 31 03:24:21.583849 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.583809 4854 scope.go:122] "RemoveContainer" containerID="714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e" Oct 31 03:24:21.584093 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:21.584074 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e\": container with ID starting with 714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e not found: ID does not exist" containerID="714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e" Oct 31 03:24:21.584158 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.584103 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e"} err="failed to get container status \"714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e\": rpc error: code = NotFound desc = could not find container \"714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e\": container with ID starting with 714517d51de2422af2b6e9133c40b6c86425c456f2dabe7d85a01d8aa845e93e not found: ID does not exist" Oct 31 03:24:21.585865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.585843 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf"] Oct 31 03:24:21.589954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:21.589934 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a2bec7-openshift-default-c6c6444c-7rlkf"] Oct 31 03:24:22.456593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:22.456563 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" path="/var/lib/kubelet/pods/9661d7af-7aba-4d48-bf9a-1f97f804b29a/volumes" Oct 31 03:24:25.743084 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:25.742661 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:29.516850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.516828 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:24:29.700319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.694818 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume\") pod \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " Oct 31 03:24:29.700319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.695184 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7\" (UniqueName: \"kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7\") pod \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\" (UID: \"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc\") " Oct 31 03:24:29.700319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.697774 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume" pod "1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" (UID: "1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:29.711284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.709329 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:29.740990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.739655 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7" pod "1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" (UID: "1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc"). InnerVolumeSpecName "kube-api-access-wbbf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:29.757469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.757433 4854 generic.go:356] "Generic (PLEG): container finished" podID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerID="455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91" exitCode=0 Oct 31 03:24:29.757699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.757683 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" Oct 31 03:24:29.757960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.757617 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" event={"ID":"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc","Type":"ContainerDied","Data":"455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91"} Oct 31 03:24:29.758139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.758117 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8" event={"ID":"1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc","Type":"ContainerDied","Data":"aceb6e2a04910d7f3f5d4c893203c57ae2bf5a3c7bdb7acea742b0b8134b2486"} Oct 31 03:24:29.758297 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.758280 4854 scope.go:122] "RemoveContainer" containerID="455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91" Oct 31 03:24:29.772945 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.772877 4854 scope.go:122] "RemoveContainer" containerID="455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91" Oct 31 03:24:29.773306 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:29.773284 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91\": container with ID starting with 455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91 not found: ID does not exist" containerID="455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91" Oct 31 03:24:29.773411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.773319 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91"} err="failed to get container status \"455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91\": rpc error: code = NotFound desc = could not find container \"455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91\": container with ID starting with 455359e461ea2d05e303b4fd210d92817154f3396016b3ea178adc67ecf30c91 not found: ID does not exist" Oct 31 03:24:29.783028 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.783002 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8"] Oct 31 03:24:29.787154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.787129 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2bec7-6d8f69fd87-9fvn8"] Oct 31 03:24:29.810948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:29.810922 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbbf7\" (UniqueName: \"kubernetes.io/projected/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc-kube-api-access-wbbf7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:30.456866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:30.456829 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" path="/var/lib/kubelet/pods/1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc/volumes" Oct 31 03:24:35.717931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:35.717898 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5"] Oct 31 03:24:35.718378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:35.718114 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" containerID="cri-o://21160b265b36319842128e44bdaf0d1112f9e2926e86349ebed5aa6f5c4384b5" gracePeriod=30 Oct 31 03:24:35.720699 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:35.720679 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts"] Oct 31 03:24:35.720903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:35.720862 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" containerID="cri-o://ef4c0973f287cae12e391cb03a452592fb75b5a37955124792272a2831cbf165" gracePeriod=30 Oct 31 03:24:35.736973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:35.736949 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8"] Oct 31 03:24:35.737139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:35.737120 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" containerID="cri-o://d3b136e60b5050065e1e140a0cc28fedbfbe33d48540a93d8ef43cbaf3b59d0e" gracePeriod=30 Oct 31 03:24:36.683291 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:36.683233 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b"] Oct 31 03:24:36.683565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:36.683534 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" containerID="cri-o://d6b29f5fc51f0a458355d302c09e93fa7ae99e5a6c06dab041276d371fe5870c" gracePeriod=30 Oct 31 03:24:36.691378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:36.691350 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg"] Oct 31 03:24:36.691608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:36.691582 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" containerID="cri-o://8b345ec6dd360cb3be84738699c7070c27647ff8a8a3868fb9da9cfdde6fcf58" gracePeriod=30 Oct 31 03:24:36.775307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:36.773677 4854 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:24:36.775307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:36.773812 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:24:39.251321 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:39.251279 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:24:40.875779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:40.875331 4854 generic.go:356] "Generic (PLEG): container finished" podID="3bb6194f-74ea-4818-bea2-fea075d97173" containerID="d3b136e60b5050065e1e140a0cc28fedbfbe33d48540a93d8ef43cbaf3b59d0e" exitCode=0 Oct 31 03:24:40.875779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:40.875463 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" event={"ID":"3bb6194f-74ea-4818-bea2-fea075d97173","Type":"ContainerDied","Data":"d3b136e60b5050065e1e140a0cc28fedbfbe33d48540a93d8ef43cbaf3b59d0e"} Oct 31 03:24:40.885467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:40.881640 4854 generic.go:356] "Generic (PLEG): container finished" podID="07765218-86e7-419d-8ae8-a981d9b47062" containerID="21160b265b36319842128e44bdaf0d1112f9e2926e86349ebed5aa6f5c4384b5" exitCode=0 Oct 31 03:24:40.885467 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:40.881756 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" event={"ID":"07765218-86e7-419d-8ae8-a981d9b47062","Type":"ContainerDied","Data":"21160b265b36319842128e44bdaf0d1112f9e2926e86349ebed5aa6f5c4384b5"} Oct 31 03:24:40.906018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:40.905984 4854 generic.go:356] "Generic (PLEG): container finished" podID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerID="ef4c0973f287cae12e391cb03a452592fb75b5a37955124792272a2831cbf165" exitCode=0 Oct 31 03:24:40.906370 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:40.906080 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" event={"ID":"f54ac6d7-4e3c-4438-adaf-f0998762ad67","Type":"ContainerDied","Data":"ef4c0973f287cae12e391cb03a452592fb75b5a37955124792272a2831cbf165"} Oct 31 03:24:41.028094 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.028059 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:24:41.032393 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032348 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032488 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032563 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032593 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032634 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032666 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032789 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032848 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.032908 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032906 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket\") pod \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\" (UID: \"f54ac6d7-4e3c-4438-adaf-f0998762ad67\") " Oct 31 03:24:41.033137 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.032919 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.033137 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.033123 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.033497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.033477 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.033568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.033503 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.033568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.033539 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.033687 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.033640 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.033747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.033694 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.036240 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.036205 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.037943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.036940 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "kube-api-access-6vrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.039046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.039021 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.040305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.040281 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo" pod "f54ac6d7-4e3c-4438-adaf-f0998762ad67" (UID: "f54ac6d7-4e3c-4438-adaf-f0998762ad67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.089242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.089216 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:24:41.103803 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.103781 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:24:41.134636 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134611 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.134821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134644 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.134821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134691 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.134821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134734 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.134821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134818 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134853 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134903 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134943 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134963 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134966 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.134992 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135057 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135020 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135073 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135112 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135154 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135180 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135216 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135264 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket\") pod \"3bb6194f-74ea-4818-bea2-fea075d97173\" (UID: \"3bb6194f-74ea-4818-bea2-fea075d97173\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135338 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135491 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135373 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.135898 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135652 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.135898 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135739 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.135898 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135849 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.135898 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.135866 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s\") pod \"07765218-86e7-419d-8ae8-a981d9b47062\" (UID: \"07765218-86e7-419d-8ae8-a981d9b47062\") " Oct 31 03:24:41.136093 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136059 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.136554 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136527 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.136681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136610 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136634 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136649 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136663 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136681 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136677 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136694 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136717 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrcb\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-kube-api-access-6vrcb\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136753 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136766 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54ac6d7-4e3c-4438-adaf-f0998762ad67-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136779 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54ac6d7-4e3c-4438-adaf-f0998762ad67-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136793 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07765218-86e7-419d-8ae8-a981d9b47062-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136806 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136819 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.136992 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.136834 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.138273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.137915 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.138273 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.138223 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.139500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.139472 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.141122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.141097 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.143781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.143755 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.143892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.143846 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.143892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.143853 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.144008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.143913 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q" pod "3bb6194f-74ea-4818-bea2-fea075d97173" (UID: "3bb6194f-74ea-4818-bea2-fea075d97173"). InnerVolumeSpecName "kube-api-access-4vh7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.144008 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.143957 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.147109 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.147086 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s" pod "07765218-86e7-419d-8ae8-a981d9b47062" (UID: "07765218-86e7-419d-8ae8-a981d9b47062"). InnerVolumeSpecName "kube-api-access-h7p6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.238087 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238048 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238090 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7p6s\" (UniqueName: \"kubernetes.io/projected/07765218-86e7-419d-8ae8-a981d9b47062-kube-api-access-h7p6s\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238109 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb6194f-74ea-4818-bea2-fea075d97173-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238126 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07765218-86e7-419d-8ae8-a981d9b47062-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238141 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238157 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238171 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb6194f-74ea-4818-bea2-fea075d97173-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238185 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238198 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb6194f-74ea-4818-bea2-fea075d97173-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238210 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vh7q\" (UniqueName: \"kubernetes.io/projected/3bb6194f-74ea-4818-bea2-fea075d97173-kube-api-access-4vh7q\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.238224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.238223 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07765218-86e7-419d-8ae8-a981d9b47062-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.924066 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" event={"ID":"3bb6194f-74ea-4818-bea2-fea075d97173","Type":"ContainerDied","Data":"47d145a00e99ec341ee947208ce20989554ec136e33c490be478c815dc189e25"} Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.924141 4854 scope.go:122] "RemoveContainer" containerID="d3b136e60b5050065e1e140a0cc28fedbfbe33d48540a93d8ef43cbaf3b59d0e" Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.924353 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8" Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.932885 4854 generic.go:356] "Generic (PLEG): container finished" podID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerID="8b345ec6dd360cb3be84738699c7070c27647ff8a8a3868fb9da9cfdde6fcf58" exitCode=0 Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.933008 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" event={"ID":"3597b530-55e8-43f0-ae5f-bc83519f7137","Type":"ContainerDied","Data":"8b345ec6dd360cb3be84738699c7070c27647ff8a8a3868fb9da9cfdde6fcf58"} Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.935049 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" event={"ID":"07765218-86e7-419d-8ae8-a981d9b47062","Type":"ContainerDied","Data":"d11ecca8131feba441cee32f91334051454f218e70b0b4eefd0dc571a8c2a9c8"} Oct 31 03:24:41.936004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.935153 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5" Oct 31 03:24:41.941569 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.941545 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" event={"ID":"f54ac6d7-4e3c-4438-adaf-f0998762ad67","Type":"ContainerDied","Data":"91eecdfecf9c7c45b0d6b16e832246fad98135b3ffffccff2ab1df9f6957c8be"} Oct 31 03:24:41.941684 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.941646 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts" Oct 31 03:24:41.951608 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.951554 4854 generic.go:356] "Generic (PLEG): container finished" podID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerID="d6b29f5fc51f0a458355d302c09e93fa7ae99e5a6c06dab041276d371fe5870c" exitCode=0 Oct 31 03:24:41.951701 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.951642 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" event={"ID":"78ebd920-0c0a-4cee-b2ef-8b307ce014f3","Type":"ContainerDied","Data":"d6b29f5fc51f0a458355d302c09e93fa7ae99e5a6c06dab041276d371fe5870c"} Oct 31 03:24:41.954929 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.954907 4854 scope.go:122] "RemoveContainer" containerID="21160b265b36319842128e44bdaf0d1112f9e2926e86349ebed5aa6f5c4384b5" Oct 31 03:24:41.972595 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.972577 4854 scope.go:122] "RemoveContainer" containerID="ef4c0973f287cae12e391cb03a452592fb75b5a37955124792272a2831cbf165" Oct 31 03:24:41.977707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.977687 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:24:41.979006 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.978985 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8"] Oct 31 03:24:41.984199 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.984179 4854 scope.go:122] "RemoveContainer" containerID="7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674" Oct 31 03:24:41.986850 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.986816 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-mcfv8"] Oct 31 03:24:41.995939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.995918 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts"] Oct 31 03:24:41.999856 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:41.999837 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tllts"] Oct 31 03:24:42.005178 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:42.005159 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674\": container with ID starting with 7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674 not found: ID does not exist" containerID="7646e7b8919e964185e7321875ed3e46979aaa89b7e3f0aef4d3aa552ab3a674" Oct 31 03:24:42.008621 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.008598 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:24:42.008873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.008859 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5"] Oct 31 03:24:42.013406 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.013385 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-l8gl5"] Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.052736 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.052823 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.052854 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.052904 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.052937 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.052972 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053064 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053008 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053091 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053157 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053200 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053225 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053311 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053352 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053380 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053408 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053474 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo\") pod \"3597b530-55e8-43f0-ae5f-bc83519f7137\" (UID: \"3597b530-55e8-43f0-ae5f-bc83519f7137\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053511 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.053662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.053546 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert\") pod \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\" (UID: \"78ebd920-0c0a-4cee-b2ef-8b307ce014f3\") " Oct 31 03:24:42.056457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.054398 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.056457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.054713 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.056457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.054901 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:42.056457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.055703 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.056457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.055899 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.056457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.056198 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.060339 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.060357 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.060900 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "kube-api-access-ht5f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.061048 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.062981 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.063039 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.063104 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.063083 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.065582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.065547 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.065582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.065553 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:42.065582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.065562 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.065582 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.065549 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo" pod "78ebd920-0c0a-4cee-b2ef-8b307ce014f3" (UID: "78ebd920-0c0a-4cee-b2ef-8b307ce014f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:42.067009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.066980 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm" pod "3597b530-55e8-43f0-ae5f-bc83519f7137" (UID: "3597b530-55e8-43f0-ae5f-bc83519f7137"). InnerVolumeSpecName "kube-api-access-hx9mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.167512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167436 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167472 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167485 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167500 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167514 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167571 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167587 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5f6\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-kube-api-access-ht5f6\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167604 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167618 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3597b530-55e8-43f0-ae5f-bc83519f7137-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167632 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167684 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167699 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9mm\" (UniqueName: \"kubernetes.io/projected/3597b530-55e8-43f0-ae5f-bc83519f7137-kube-api-access-hx9mm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167711 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167725 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3597b530-55e8-43f0-ae5f-bc83519f7137-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167741 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167757 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167773 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78ebd920-0c0a-4cee-b2ef-8b307ce014f3-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.167872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.167788 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3597b530-55e8-43f0-ae5f-bc83519f7137-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.200719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.200680 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg"] Oct 31 03:24:42.205586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.205524 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww"] Oct 31 03:24:42.458043 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.457967 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07765218-86e7-419d-8ae8-a981d9b47062" path="/var/lib/kubelet/pods/07765218-86e7-419d-8ae8-a981d9b47062/volumes" Oct 31 03:24:42.458542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.458527 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" path="/var/lib/kubelet/pods/3bb6194f-74ea-4818-bea2-fea075d97173/volumes" Oct 31 03:24:42.458939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.458927 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" path="/var/lib/kubelet/pods/f54ac6d7-4e3c-4438-adaf-f0998762ad67/volumes" Oct 31 03:24:42.982272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.981592 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" event={"ID":"3597b530-55e8-43f0-ae5f-bc83519f7137","Type":"ContainerDied","Data":"505c2d47e8e1b191ce3b38df6bdf8581ccc66675325e19244deefcd73dbcb241"} Oct 31 03:24:42.982272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.981683 4854 scope.go:122] "RemoveContainer" containerID="8b345ec6dd360cb3be84738699c7070c27647ff8a8a3868fb9da9cfdde6fcf58" Oct 31 03:24:42.982272 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:42.981903 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg" Oct 31 03:24:43.019236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.014680 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" event={"ID":"78ebd920-0c0a-4cee-b2ef-8b307ce014f3","Type":"ContainerDied","Data":"31e51f33528dda0a35ba2acc377f423a5748fd695ba8e4a0433cced2a3c4d794"} Oct 31 03:24:43.019236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.014746 4854 scope.go:122] "RemoveContainer" containerID="d6b29f5fc51f0a458355d302c09e93fa7ae99e5a6c06dab041276d371fe5870c" Oct 31 03:24:43.019236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.014865 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b" Oct 31 03:24:43.019236 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.017343 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg"] Oct 31 03:24:43.021200 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.021167 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-vkdqg"] Oct 31 03:24:43.036512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.036488 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b"] Oct 31 03:24:43.042319 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:43.042295 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-bzm6b"] Oct 31 03:24:44.456659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:44.456622 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" path="/var/lib/kubelet/pods/3597b530-55e8-43f0-ae5f-bc83519f7137/volumes" Oct 31 03:24:44.457078 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:44.457063 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" path="/var/lib/kubelet/pods/78ebd920-0c0a-4cee-b2ef-8b307ce014f3/volumes" Oct 31 03:24:47.418841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:47.418804 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t"] Oct 31 03:24:47.419366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:47.419126 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" podUID="867896b1-871f-47e4-8a01-a283fe3e2810" containerName="maas-api" containerID="cri-o://176ac9f3c15622e219eb01c016d7641d3bb3adc8ac09a8b17347f7c9690e4482" gracePeriod=30 Oct 31 03:24:48.052013 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.051986 4854 generic.go:356] "Generic (PLEG): container finished" podID="867896b1-871f-47e4-8a01-a283fe3e2810" containerID="176ac9f3c15622e219eb01c016d7641d3bb3adc8ac09a8b17347f7c9690e4482" exitCode=0 Oct 31 03:24:48.052151 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.052063 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" event={"ID":"867896b1-871f-47e4-8a01-a283fe3e2810","Type":"ContainerDied","Data":"176ac9f3c15622e219eb01c016d7641d3bb3adc8ac09a8b17347f7c9690e4482"} Oct 31 03:24:48.187174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.187148 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:24:48.250709 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.250676 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls\") pod \"867896b1-871f-47e4-8a01-a283fe3e2810\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " Oct 31 03:24:48.250860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.250773 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca\") pod \"867896b1-871f-47e4-8a01-a283fe3e2810\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " Oct 31 03:24:48.250913 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.250873 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr\" (UniqueName: \"kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr\") pod \"867896b1-871f-47e4-8a01-a283fe3e2810\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " Oct 31 03:24:48.250999 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.250979 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs\") pod \"867896b1-871f-47e4-8a01-a283fe3e2810\" (UID: \"867896b1-871f-47e4-8a01-a283fe3e2810\") " Oct 31 03:24:48.251482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.251157 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca" pod "867896b1-871f-47e4-8a01-a283fe3e2810" (UID: "867896b1-871f-47e4-8a01-a283fe3e2810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:48.251482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.251458 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867896b1-871f-47e4-8a01-a283fe3e2810-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:48.253607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.253578 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr" pod "867896b1-871f-47e4-8a01-a283fe3e2810" (UID: "867896b1-871f-47e4-8a01-a283fe3e2810"). InnerVolumeSpecName "kube-api-access-zl6rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:48.253703 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.253647 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs" pod "867896b1-871f-47e4-8a01-a283fe3e2810" (UID: "867896b1-871f-47e4-8a01-a283fe3e2810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:48.253758 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.253700 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls" pod "867896b1-871f-47e4-8a01-a283fe3e2810" (UID: "867896b1-871f-47e4-8a01-a283fe3e2810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:48.352349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.352314 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:48.352349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.352342 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl6rr\" (UniqueName: \"kubernetes.io/projected/867896b1-871f-47e4-8a01-a283fe3e2810-kube-api-access-zl6rr\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:48.352349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:48.352353 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867896b1-871f-47e4-8a01-a283fe3e2810-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:49.059612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:49.059582 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" Oct 31 03:24:49.060033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:49.059618 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t" event={"ID":"867896b1-871f-47e4-8a01-a283fe3e2810","Type":"ContainerDied","Data":"58a24b2bd2c460b95a71afa5983b167029db0a2aa1c7d4c73259c7361f770fb0"} Oct 31 03:24:49.060033 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:49.059683 4854 scope.go:122] "RemoveContainer" containerID="176ac9f3c15622e219eb01c016d7641d3bb3adc8ac09a8b17347f7c9690e4482" Oct 31 03:24:49.080267 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:49.080228 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t"] Oct 31 03:24:49.089149 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:49.089120 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b79c13-7974f85876-7dv7t"] Oct 31 03:24:50.455885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:50.455852 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867896b1-871f-47e4-8a01-a283fe3e2810" path="/var/lib/kubelet/pods/867896b1-871f-47e4-8a01-a283fe3e2810/volumes" Oct 31 03:24:56.268623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:56.268581 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn"] Oct 31 03:24:56.269018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:56.268874 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" containerID="cri-o://b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754" gracePeriod=30 Oct 31 03:24:57.227518 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.227460 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" containerID="cri-o://671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9" gracePeriod=45 Oct 31 03:24:57.227809 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.227783 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" containerID="cri-o://4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b" gracePeriod=45 Oct 31 03:24:57.505077 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.505053 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:24:57.541124 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.541096 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:24:57.597998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.597971 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache\") pod \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " Oct 31 03:24:57.598192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.598047 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm\" (UniqueName: \"kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm\") pod \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " Oct 31 03:24:57.598192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.598121 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm\") pod \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " Oct 31 03:24:57.598377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.598199 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs\") pod \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " Oct 31 03:24:57.598377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.598352 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir\") pod \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " Oct 31 03:24:57.598486 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.598442 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home\") pod \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\" (UID: \"1c17918e-87b0-4d8c-88e5-d072a50b9f59\") " Oct 31 03:24:57.599290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.598986 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir" pod "1c17918e-87b0-4d8c-88e5-d072a50b9f59" (UID: "1c17918e-87b0-4d8c-88e5-d072a50b9f59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.599290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.599123 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home" pod "1c17918e-87b0-4d8c-88e5-d072a50b9f59" (UID: "1c17918e-87b0-4d8c-88e5-d072a50b9f59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.599290 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.599225 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.599616 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.599587 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache" pod "1c17918e-87b0-4d8c-88e5-d072a50b9f59" (UID: "1c17918e-87b0-4d8c-88e5-d072a50b9f59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.601112 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.601089 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm" pod "1c17918e-87b0-4d8c-88e5-d072a50b9f59" (UID: "1c17918e-87b0-4d8c-88e5-d072a50b9f59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.601222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.601139 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm" pod "1c17918e-87b0-4d8c-88e5-d072a50b9f59" (UID: "1c17918e-87b0-4d8c-88e5-d072a50b9f59"). InnerVolumeSpecName "kube-api-access-cw6mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:57.601989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.601961 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs" pod "1c17918e-87b0-4d8c-88e5-d072a50b9f59" (UID: "1c17918e-87b0-4d8c-88e5-d072a50b9f59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:57.699935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.699903 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir\") pod \"327587c2-603d-4c5e-8c05-f06f49e634f9\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " Oct 31 03:24:57.700091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.699986 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm\") pod \"327587c2-603d-4c5e-8c05-f06f49e634f9\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " Oct 31 03:24:57.700091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700029 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs\") pod \"327587c2-603d-4c5e-8c05-f06f49e634f9\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " Oct 31 03:24:57.700091 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700053 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache\") pod \"327587c2-603d-4c5e-8c05-f06f49e634f9\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " Oct 31 03:24:57.700305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700125 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7\" (UniqueName: \"kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7\") pod \"327587c2-603d-4c5e-8c05-f06f49e634f9\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " Oct 31 03:24:57.700305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700165 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir" pod "327587c2-603d-4c5e-8c05-f06f49e634f9" (UID: "327587c2-603d-4c5e-8c05-f06f49e634f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.700305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700197 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home\") pod \"327587c2-603d-4c5e-8c05-f06f49e634f9\" (UID: \"327587c2-603d-4c5e-8c05-f06f49e634f9\") " Oct 31 03:24:57.700482 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700356 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache" pod "327587c2-603d-4c5e-8c05-f06f49e634f9" (UID: "327587c2-603d-4c5e-8c05-f06f49e634f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.700546 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700490 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home" pod "327587c2-603d-4c5e-8c05-f06f49e634f9" (UID: "327587c2-603d-4c5e-8c05-f06f49e634f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.700670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700653 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700671 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c17918e-87b0-4d8c-88e5-d072a50b9f59-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700679 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700689 4854 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-tmp-dir\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700696 4854 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-home\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700707 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c17918e-87b0-4d8c-88e5-d072a50b9f59-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700723 4854 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-model-cache\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.700792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.700737 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6mm\" (UniqueName: \"kubernetes.io/projected/1c17918e-87b0-4d8c-88e5-d072a50b9f59-kube-api-access-cw6mm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.702102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.702085 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm" pod "327587c2-603d-4c5e-8c05-f06f49e634f9" (UID: "327587c2-603d-4c5e-8c05-f06f49e634f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.702205 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.702194 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs" pod "327587c2-603d-4c5e-8c05-f06f49e634f9" (UID: "327587c2-603d-4c5e-8c05-f06f49e634f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:57.702415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.702399 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7" pod "327587c2-603d-4c5e-8c05-f06f49e634f9" (UID: "327587c2-603d-4c5e-8c05-f06f49e634f9"). InnerVolumeSpecName "kube-api-access-xtgr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:57.802115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.802084 4854 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327587c2-603d-4c5e-8c05-f06f49e634f9-dshm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.802115 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.802110 4854 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327587c2-603d-4c5e-8c05-f06f49e634f9-tls-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.802345 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:57.802122 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtgr7\" (UniqueName: \"kubernetes.io/projected/327587c2-603d-4c5e-8c05-f06f49e634f9-kube-api-access-xtgr7\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.119872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.119777 4854 generic.go:356] "Generic (PLEG): container finished" podID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerID="671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9" exitCode=0 Oct 31 03:24:58.119872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.119826 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" Oct 31 03:24:58.120114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.119869 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" event={"ID":"327587c2-603d-4c5e-8c05-f06f49e634f9","Type":"ContainerDied","Data":"671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9"} Oct 31 03:24:58.120114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.119909 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg" event={"ID":"327587c2-603d-4c5e-8c05-f06f49e634f9","Type":"ContainerDied","Data":"14529a6c86d5ea283ab92601309d46d66c877288a332abe4fc2fdf78f88a5e20"} Oct 31 03:24:58.120114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.119930 4854 scope.go:122] "RemoveContainer" containerID="671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9" Oct 31 03:24:58.121447 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.121426 4854 generic.go:356] "Generic (PLEG): container finished" podID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerID="4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b" exitCode=0 Oct 31 03:24:58.121598 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.121466 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" Oct 31 03:24:58.121598 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.121529 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" event={"ID":"1c17918e-87b0-4d8c-88e5-d072a50b9f59","Type":"ContainerDied","Data":"4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b"} Oct 31 03:24:58.121598 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.121563 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" event={"ID":"1c17918e-87b0-4d8c-88e5-d072a50b9f59","Type":"ContainerDied","Data":"554d579a087953bf812c676649cb06d4032e11b44e9878baf30a64bc2c269504"} Oct 31 03:24:58.129568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.129511 4854 scope.go:122] "RemoveContainer" containerID="671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9" Oct 31 03:24:58.129852 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:58.129828 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9\": container with ID starting with 671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9 not found: ID does not exist" containerID="671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9" Oct 31 03:24:58.129968 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.129857 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9"} err="failed to get container status \"671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9\": rpc error: code = NotFound desc = could not find container \"671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9\": container with ID starting with 671ee84c9df5955ebfa7650d728874dda359998ff61b50e04f7c635cbb47edc9 not found: ID does not exist" Oct 31 03:24:58.129968 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.129872 4854 scope.go:122] "RemoveContainer" containerID="4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b" Oct 31 03:24:58.138563 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.138546 4854 scope.go:122] "RemoveContainer" containerID="4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b" Oct 31 03:24:58.138854 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:58.138835 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b\": container with ID starting with 4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b not found: ID does not exist" containerID="4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b" Oct 31 03:24:58.138943 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.138859 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b"} err="failed to get container status \"4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b\": rpc error: code = NotFound desc = could not find container \"4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b\": container with ID starting with 4f84b703441f8d563735903c7f2194c7ed9632060affee5b5f1fe707ae28cd9b not found: ID does not exist" Oct 31 03:24:58.143800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.143779 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg"] Oct 31 03:24:58.147998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.147979 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-2-w1-924ee5-kserve-558c887945-4qgmg"] Oct 31 03:24:58.157807 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.157776 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww"] Oct 31 03:24:58.159900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.159879 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww"] Oct 31 03:24:58.366580 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.366547 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx"] Oct 31 03:24:58.366792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.366769 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerName="discovery" containerID="cri-o://2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377" gracePeriod=30 Oct 31 03:24:58.456043 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.456010 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" path="/var/lib/kubelet/pods/1c17918e-87b0-4d8c-88e5-d072a50b9f59/volumes" Oct 31 03:24:58.456481 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.456467 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" path="/var/lib/kubelet/pods/327587c2-603d-4c5e-8c05-f06f49e634f9/volumes" Oct 31 03:24:58.457840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.457818 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/distinct-w1-924ee5-kserve-84f9644db-kfhww" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:24:58.652210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.652184 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:24:58.812894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.812868 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.812923 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.812959 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.812982 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813049 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.813011 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.813068 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.813133 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts\") pod \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\" (UID: \"6e17e36c-cf1f-485f-a02f-a2c57b1f699f\") " Oct 31 03:24:58.813654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.813428 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:58.813654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.813571 4854 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-ca-configmap\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.815637 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.815602 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "kube-api-access-zr6tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:58.815637 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.815621 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:58.815802 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.815707 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:58.815903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.815887 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:58.815973 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.815900 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:58.816032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.815995 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts" pod "6e17e36c-cf1f-485f-a02f-a2c57b1f699f" (UID: "6e17e36c-cf1f-485f-a02f-a2c57b1f699f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:58.914962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.914870 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6tx\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-kube-api-access-zr6tx\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.914962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.914901 4854 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-cacerts\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.914962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.914917 4854 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-csr-dns-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.914962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.914930 4854 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-local-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.914962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.914942 4854 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-kubeconfig\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:58.914962 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:58.914956 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17e36c-cf1f-485f-a02f-a2c57b1f699f-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:59.130613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.130583 4854 generic.go:356] "Generic (PLEG): container finished" podID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerID="2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377" exitCode=0 Oct 31 03:24:59.130781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.130645 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" Oct 31 03:24:59.130781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.130663 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" event={"ID":"6e17e36c-cf1f-485f-a02f-a2c57b1f699f","Type":"ContainerDied","Data":"2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377"} Oct 31 03:24:59.130781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.130728 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx" event={"ID":"6e17e36c-cf1f-485f-a02f-a2c57b1f699f","Type":"ContainerDied","Data":"c0868c111a258e1285031d08155345f715b9b2ea7108d88b379135946875b247"} Oct 31 03:24:59.130781 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.130756 4854 scope.go:122] "RemoveContainer" containerID="2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377" Oct 31 03:24:59.141731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.141648 4854 scope.go:122] "RemoveContainer" containerID="2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377" Oct 31 03:24:59.142027 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:24:59.141993 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377\": container with ID starting with 2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377 not found: ID does not exist" containerID="2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377" Oct 31 03:24:59.142117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.142041 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377"} err="failed to get container status \"2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377\": rpc error: code = NotFound desc = could not find container \"2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377\": container with ID starting with 2334a27b57e6712b4bbca2cb4e152eb84e87500ccc7cbfe3e8afddcb97cfd377 not found: ID does not exist" Oct 31 03:24:59.155413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.155386 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx"] Oct 31 03:24:59.157443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:24:59.157424 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dmhxx"] Oct 31 03:25:00.457063 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:00.457028 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" path="/var/lib/kubelet/pods/6e17e36c-cf1f-485f-a02f-a2c57b1f699f/volumes" Oct 31 03:25:01.519600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.519575 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:25:01.664988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.664906 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665003 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665045 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665084 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665122 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665418 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665177 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665418 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665210 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665418 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665241 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665703 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket\") pod \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\" (UID: \"e93ff988-5bc5-495c-99fa-2f8bdbe432b6\") " Oct 31 03:25:01.665736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665324 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.666079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665492 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.666079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665558 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.666079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.665633 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:01.666287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.666080 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.666287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.666190 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.666287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.666204 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.666287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.666213 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.666287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.666221 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.666287 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.666229 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.667637 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.667613 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "kube-api-access-szqvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:01.667637 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.667620 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:01.667909 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.667884 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:01.667909 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.667889 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy" pod "e93ff988-5bc5-495c-99fa-2f8bdbe432b6" (UID: "e93ff988-5bc5-495c-99fa-2f8bdbe432b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.767795 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.767768 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.767795 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.767795 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.767944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.767804 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.767944 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:01.767814 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szqvm\" (UniqueName: \"kubernetes.io/projected/e93ff988-5bc5-495c-99fa-2f8bdbe432b6-kube-api-access-szqvm\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:02.150722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.150691 4854 generic.go:356] "Generic (PLEG): container finished" podID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerID="b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754" exitCode=0 Oct 31 03:25:02.150900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.150734 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" event={"ID":"e93ff988-5bc5-495c-99fa-2f8bdbe432b6","Type":"ContainerDied","Data":"b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754"} Oct 31 03:25:02.150900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.150754 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" Oct 31 03:25:02.150900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.150787 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn" event={"ID":"e93ff988-5bc5-495c-99fa-2f8bdbe432b6","Type":"ContainerDied","Data":"67adff6ed8293c3133a08cba8a2164f46e0075027ebb14c52f8a9cfb35c20f15"} Oct 31 03:25:02.150900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.150818 4854 scope.go:122] "RemoveContainer" containerID="b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754" Oct 31 03:25:02.161586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.161565 4854 scope.go:122] "RemoveContainer" containerID="ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465" Oct 31 03:25:02.170065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.170001 4854 scope.go:122] "RemoveContainer" containerID="b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754" Oct 31 03:25:02.170410 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:02.170372 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754\": container with ID starting with b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754 not found: ID does not exist" containerID="b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754" Oct 31 03:25:02.170524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.170421 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754"} err="failed to get container status \"b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754\": rpc error: code = NotFound desc = could not find container \"b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754\": container with ID starting with b5ebcac61007eb69a17fdf3298a1636457315447bbae1420bdc36aa7e907c754 not found: ID does not exist" Oct 31 03:25:02.170524 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.170444 4854 scope.go:122] "RemoveContainer" containerID="ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465" Oct 31 03:25:02.170823 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:02.170798 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465\": container with ID starting with ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465 not found: ID does not exist" containerID="ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465" Oct 31 03:25:02.170884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.170833 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465"} err="failed to get container status \"ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465\": rpc error: code = NotFound desc = could not find container \"ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465\": container with ID starting with ff1048e93b82847a66cf2142d603465b4a8b2b18f8e243c4148a204e91d56465 not found: ID does not exist" Oct 31 03:25:02.173095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.173075 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn"] Oct 31 03:25:02.178610 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.178587 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-cgrrn"] Oct 31 03:25:02.456023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:02.455959 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" path="/var/lib/kubelet/pods/e93ff988-5bc5-495c-99fa-2f8bdbe432b6/volumes" Oct 31 03:25:06.609941 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:06.609907 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd"] Oct 31 03:25:06.610647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:06.610132 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" podUID="9bf4c889-9afb-4325-b343-ec7fb2251357" containerName="maas-api" containerID="cri-o://4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50" gracePeriod=30 Oct 31 03:25:06.975409 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:06.975385 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:25:07.019798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.019742 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx"] Oct 31 03:25:07.023690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.023625 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls\") pod \"9bf4c889-9afb-4325-b343-ec7fb2251357\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " Oct 31 03:25:07.023690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.023801 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs\") pod \"9bf4c889-9afb-4325-b343-ec7fb2251357\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " Oct 31 03:25:07.023690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.023858 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca\") pod \"9bf4c889-9afb-4325-b343-ec7fb2251357\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " Oct 31 03:25:07.023690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.023893 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298\" (UniqueName: \"kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298\") pod \"9bf4c889-9afb-4325-b343-ec7fb2251357\" (UID: \"9bf4c889-9afb-4325-b343-ec7fb2251357\") " Oct 31 03:25:07.023690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.024507 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca" pod "9bf4c889-9afb-4325-b343-ec7fb2251357" (UID: "9bf4c889-9afb-4325-b343-ec7fb2251357"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:07.042788 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.042749 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs" pod "9bf4c889-9afb-4325-b343-ec7fb2251357" (UID: "9bf4c889-9afb-4325-b343-ec7fb2251357"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:07.044690 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.044662 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls" pod "9bf4c889-9afb-4325-b343-ec7fb2251357" (UID: "9bf4c889-9afb-4325-b343-ec7fb2251357"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:07.045901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.045876 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298" pod "9bf4c889-9afb-4325-b343-ec7fb2251357" (UID: "9bf4c889-9afb-4325-b343-ec7fb2251357"). InnerVolumeSpecName "kube-api-access-fr298". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:07.125765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.125730 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:07.125765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.125758 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bf4c889-9afb-4325-b343-ec7fb2251357-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:07.125765 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.125767 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr298\" (UniqueName: \"kubernetes.io/projected/9bf4c889-9afb-4325-b343-ec7fb2251357-kube-api-access-fr298\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:07.126077 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.125777 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bf4c889-9afb-4325-b343-ec7fb2251357-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:07.190114 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.190078 4854 generic.go:356] "Generic (PLEG): container finished" podID="9bf4c889-9afb-4325-b343-ec7fb2251357" containerID="4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50" exitCode=0 Oct 31 03:25:07.190328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.190129 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" Oct 31 03:25:07.190328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.190165 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" event={"ID":"9bf4c889-9afb-4325-b343-ec7fb2251357","Type":"ContainerDied","Data":"4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50"} Oct 31 03:25:07.190328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.190229 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd" event={"ID":"9bf4c889-9afb-4325-b343-ec7fb2251357","Type":"ContainerDied","Data":"bc9915c2c2cd672f2084802798abfa3a0fada385c3f992abb418d71d6218c24c"} Oct 31 03:25:07.190328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.190300 4854 scope.go:122] "RemoveContainer" containerID="4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50" Oct 31 03:25:07.200542 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.200528 4854 scope.go:122] "RemoveContainer" containerID="4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50" Oct 31 03:25:07.200793 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:07.200773 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50\": container with ID starting with 4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50 not found: ID does not exist" containerID="4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50" Oct 31 03:25:07.200859 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.200800 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50"} err="failed to get container status \"4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50\": rpc error: code = NotFound desc = could not find container \"4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50\": container with ID starting with 4cb8f5db0c344baf89555ef96ab2620e2d103cb64c2e6f3af4f47d268115db50 not found: ID does not exist" Oct 31 03:25:07.211780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.211756 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd"] Oct 31 03:25:07.215797 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:07.215775 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-924ee5-55f5cb6b46-sv7xd"] Oct 31 03:25:08.455609 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:08.455577 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf4c889-9afb-4325-b343-ec7fb2251357" path="/var/lib/kubelet/pods/9bf4c889-9afb-4325-b343-ec7fb2251357/volumes" Oct 31 03:25:09.250192 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:09.250157 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:25:10.049460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.049425 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs"] Oct 31 03:25:10.049965 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.049632 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" podUID="390be04b-2767-42a6-bed1-4466d0ca43f0" containerName="maas-api" containerID="cri-o://d982a94b53f7ec81e9109cb9835e3fcd519df72ce5bf93859c960266a294c9dc" gracePeriod=30 Oct 31 03:25:10.221320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.221281 4854 generic.go:356] "Generic (PLEG): container finished" podID="390be04b-2767-42a6-bed1-4466d0ca43f0" containerID="d982a94b53f7ec81e9109cb9835e3fcd519df72ce5bf93859c960266a294c9dc" exitCode=0 Oct 31 03:25:10.221494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.221353 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" event={"ID":"390be04b-2767-42a6-bed1-4466d0ca43f0","Type":"ContainerDied","Data":"d982a94b53f7ec81e9109cb9835e3fcd519df72ce5bf93859c960266a294c9dc"} Oct 31 03:25:10.399461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.399438 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:25:10.472213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.472181 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs\") pod \"390be04b-2767-42a6-bed1-4466d0ca43f0\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " Oct 31 03:25:10.472213 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.472214 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\") pod \"390be04b-2767-42a6-bed1-4466d0ca43f0\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " Oct 31 03:25:10.474512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.474483 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs" pod "390be04b-2767-42a6-bed1-4466d0ca43f0" (UID: "390be04b-2767-42a6-bed1-4466d0ca43f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:10.474620 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.474539 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls" pod "390be04b-2767-42a6-bed1-4466d0ca43f0" (UID: "390be04b-2767-42a6-bed1-4466d0ca43f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:10.574647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.573774 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt\" (UniqueName: \"kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt\") pod \"390be04b-2767-42a6-bed1-4466d0ca43f0\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " Oct 31 03:25:10.574647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.573849 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca\") pod \"390be04b-2767-42a6-bed1-4466d0ca43f0\" (UID: \"390be04b-2767-42a6-bed1-4466d0ca43f0\") " Oct 31 03:25:10.574647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.574426 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:10.574647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.574449 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390be04b-2767-42a6-bed1-4466d0ca43f0-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:10.574647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.574512 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca" pod "390be04b-2767-42a6-bed1-4466d0ca43f0" (UID: "390be04b-2767-42a6-bed1-4466d0ca43f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:10.578377 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.578337 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt" pod "390be04b-2767-42a6-bed1-4466d0ca43f0" (UID: "390be04b-2767-42a6-bed1-4466d0ca43f0"). InnerVolumeSpecName "kube-api-access-dxczt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:10.675310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.675213 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxczt\" (UniqueName: \"kubernetes.io/projected/390be04b-2767-42a6-bed1-4466d0ca43f0-kube-api-access-dxczt\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:10.675310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:10.675240 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390be04b-2767-42a6-bed1-4466d0ca43f0-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:11.228723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:11.228691 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" event={"ID":"390be04b-2767-42a6-bed1-4466d0ca43f0","Type":"ContainerDied","Data":"8485ea790a92ebca88d1389d7686bff04af1b27c73f0312ea647d436a229050f"} Oct 31 03:25:11.228723 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:11.228720 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs" Oct 31 03:25:11.229235 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:11.228754 4854 scope.go:122] "RemoveContainer" containerID="d982a94b53f7ec81e9109cb9835e3fcd519df72ce5bf93859c960266a294c9dc" Oct 31 03:25:11.249963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:11.249934 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs"] Oct 31 03:25:11.260054 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:11.260029 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2206e7-5d7f7d488b-4j4xs"] Oct 31 03:25:12.020716 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.020683 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" podUID="38943b33-ed6e-492b-beb8-0eb4bf8a8d81" containerName="payload-processing" containerID="cri-o://c52fa572adf9c27cf837a46e3ba61faafc67c4bc13734453b3887e874370fb2b" gracePeriod=25 Oct 31 03:25:12.244226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.244190 4854 generic.go:356] "Generic (PLEG): container finished" podID="38943b33-ed6e-492b-beb8-0eb4bf8a8d81" containerID="c52fa572adf9c27cf837a46e3ba61faafc67c4bc13734453b3887e874370fb2b" exitCode=0 Oct 31 03:25:12.244576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.244244 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" event={"ID":"38943b33-ed6e-492b-beb8-0eb4bf8a8d81","Type":"ContainerDied","Data":"c52fa572adf9c27cf837a46e3ba61faafc67c4bc13734453b3887e874370fb2b"} Oct 31 03:25:12.328900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.328871 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:25:12.391490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.391462 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv\" (UniqueName: \"kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv\") pod \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " Oct 31 03:25:12.391647 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.391534 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume\") pod \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\" (UID: \"38943b33-ed6e-492b-beb8-0eb4bf8a8d81\") " Oct 31 03:25:12.391912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.391882 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume" pod "38943b33-ed6e-492b-beb8-0eb4bf8a8d81" (UID: "38943b33-ed6e-492b-beb8-0eb4bf8a8d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:12.392042 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.392026 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:12.393825 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.393790 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv" pod "38943b33-ed6e-492b-beb8-0eb4bf8a8d81" (UID: "38943b33-ed6e-492b-beb8-0eb4bf8a8d81"). InnerVolumeSpecName "kube-api-access-89cmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:12.455070 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.455038 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390be04b-2767-42a6-bed1-4466d0ca43f0" path="/var/lib/kubelet/pods/390be04b-2767-42a6-bed1-4466d0ca43f0/volumes" Oct 31 03:25:12.493740 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:12.493645 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89cmv\" (UniqueName: \"kubernetes.io/projected/38943b33-ed6e-492b-beb8-0eb4bf8a8d81-kube-api-access-89cmv\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:13.252338 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:13.252305 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" Oct 31 03:25:13.252800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:13.252312 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx" event={"ID":"38943b33-ed6e-492b-beb8-0eb4bf8a8d81","Type":"ContainerDied","Data":"eed0719aac1aab6e474116d286f0f9cfe7753c001f5a45a19129e6f47591761d"} Oct 31 03:25:13.252800 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:13.252437 4854 scope.go:122] "RemoveContainer" containerID="c52fa572adf9c27cf837a46e3ba61faafc67c4bc13734453b3887e874370fb2b" Oct 31 03:25:13.267125 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:13.267099 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx"] Oct 31 03:25:13.271390 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:13.271365 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2206e7-d58c9777-wt4wx"] Oct 31 03:25:14.455804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:14.455771 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38943b33-ed6e-492b-beb8-0eb4bf8a8d81" path="/var/lib/kubelet/pods/38943b33-ed6e-492b-beb8-0eb4bf8a8d81/volumes" Oct 31 03:25:16.220735 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:16.220685 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4"] Oct 31 03:25:16.221348 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:16.220997 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" containerID="cri-o://d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544" gracePeriod=30 Oct 31 03:25:18.760351 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:18.760312 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94"] Oct 31 03:25:18.760842 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:18.760576 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" containerID="cri-o://0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623" gracePeriod=30 Oct 31 03:25:19.502178 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:19.502129 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:25:19.505609 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:19.505573 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:25:19.518804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:19.518769 4854 kubelet.go:1689] "Image garbage collection succeeded" Oct 31 03:25:19.914439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:19.914410 4854 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:19.914820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:19.914480 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:20.955210 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:20.955172 4854 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:20.955660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:20.955235 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:21.480067 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.480042 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:25:21.541711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541643 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.541711 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541704 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.541940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541756 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.541940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541789 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.541940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541835 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.541940 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541877 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.542171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541954 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.542171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.541994 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.542171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.542058 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo\") pod \"fba893f3-b888-41af-a479-93b4965a7813\" (UID: \"fba893f3-b888-41af-a479-93b4965a7813\") " Oct 31 03:25:21.542171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.542096 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.542171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.542116 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:21.542461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.542327 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.542770 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.542747 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.543165 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.543141 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.543607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.543581 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba893f3-b888-41af-a479-93b4965a7813-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.543736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.543611 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.543736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.543628 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.543736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.543642 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.543736 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.543660 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.544760 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.544732 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "kube-api-access-s5xmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.544952 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.544933 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:21.545122 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.545091 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.545233 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.545215 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token" pod "fba893f3-b888-41af-a479-93b4965a7813" (UID: "fba893f3-b888-41af-a479-93b4965a7813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.644792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.644764 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba893f3-b888-41af-a479-93b4965a7813-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.644792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.644788 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.644957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.644798 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xmt\" (UniqueName: \"kubernetes.io/projected/fba893f3-b888-41af-a479-93b4965a7813-kube-api-access-s5xmt\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.644957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:21.644809 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba893f3-b888-41af-a479-93b4965a7813-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:22.304868 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.304826 4854 generic.go:356] "Generic (PLEG): container finished" podID="fba893f3-b888-41af-a479-93b4965a7813" containerID="d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544" exitCode=0 Oct 31 03:25:22.305300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.304885 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" Oct 31 03:25:22.305300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.304907 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" event={"ID":"fba893f3-b888-41af-a479-93b4965a7813","Type":"ContainerDied","Data":"d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544"} Oct 31 03:25:22.305300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.304966 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4" event={"ID":"fba893f3-b888-41af-a479-93b4965a7813","Type":"ContainerDied","Data":"8d91042f1fea54ef9d7a6edde54120d01d4935c6069d4f879f03051b7d7b7cb1"} Oct 31 03:25:22.305300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.304988 4854 scope.go:122] "RemoveContainer" containerID="d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544" Oct 31 03:25:22.315860 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.315814 4854 scope.go:122] "RemoveContainer" containerID="7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a" Oct 31 03:25:22.324827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.324806 4854 scope.go:122] "RemoveContainer" containerID="d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544" Oct 31 03:25:22.325239 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:22.325205 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544\": container with ID starting with d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544 not found: ID does not exist" containerID="d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544" Oct 31 03:25:22.325362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.325274 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544"} err="failed to get container status \"d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544\": rpc error: code = NotFound desc = could not find container \"d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544\": container with ID starting with d26b948ef4d62b2de55691ab6580eca0bab0cdf9ad7f48950970fe45fb4c7544 not found: ID does not exist" Oct 31 03:25:22.325362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.325299 4854 scope.go:122] "RemoveContainer" containerID="7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a" Oct 31 03:25:22.325637 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:22.325610 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a\": container with ID starting with 7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a not found: ID does not exist" containerID="7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a" Oct 31 03:25:22.325750 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.325643 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a"} err="failed to get container status \"7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a\": rpc error: code = NotFound desc = could not find container \"7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a\": container with ID starting with 7986086310e358e3b5a010a0ec1c864d2407278e2e27d09ed14ac735aa46448a not found: ID does not exist" Oct 31 03:25:22.327709 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.327689 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4"] Oct 31 03:25:22.333300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.333277 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-zd6c4"] Oct 31 03:25:22.456328 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:22.456306 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba893f3-b888-41af-a479-93b4965a7813" path="/var/lib/kubelet/pods/fba893f3-b888-41af-a479-93b4965a7813/volumes" Oct 31 03:25:24.025466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.025414 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:25:24.170752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170721 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.170752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170756 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.170976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170804 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.170976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170836 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.170976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170901 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.170976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170932 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.170976 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.170970 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.171298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171018 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.171298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171067 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token\") pod \"7ae6cf33-76ca-40b1-9424-59dadb8de118\" (UID: \"7ae6cf33-76ca-40b1-9424-59dadb8de118\") " Oct 31 03:25:24.171557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171515 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:24.171715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171688 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.171715 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171703 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.171843 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171816 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.171900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.171865 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.173321 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.173295 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "kube-api-access-487vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.174077 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.174050 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.174166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.174080 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.174227 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.174210 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo" pod "7ae6cf33-76ca-40b1-9424-59dadb8de118" (UID: "7ae6cf33-76ca-40b1-9424-59dadb8de118"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:24.272553 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272532 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae6cf33-76ca-40b1-9424-59dadb8de118-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272555 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487vq\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-kube-api-access-487vq\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272567 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272576 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272585 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272593 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272602 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae6cf33-76ca-40b1-9424-59dadb8de118-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272610 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.272644 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.272618 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae6cf33-76ca-40b1-9424-59dadb8de118-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.317660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.317600 4854 generic.go:356] "Generic (PLEG): container finished" podID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerID="0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623" exitCode=0 Oct 31 03:25:24.317660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.317654 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" Oct 31 03:25:24.317779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.317680 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" event={"ID":"7ae6cf33-76ca-40b1-9424-59dadb8de118","Type":"ContainerDied","Data":"0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623"} Oct 31 03:25:24.317779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.317730 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94" event={"ID":"7ae6cf33-76ca-40b1-9424-59dadb8de118","Type":"ContainerDied","Data":"d42492e5a9aa1757a594f257fb5045c77ef416b20491811235c075d0a96e3661"} Oct 31 03:25:24.317779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.317751 4854 scope.go:122] "RemoveContainer" containerID="0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623" Oct 31 03:25:24.328399 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.328381 4854 scope.go:122] "RemoveContainer" containerID="0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623" Oct 31 03:25:24.328654 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:24.328632 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623\": container with ID starting with 0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623 not found: ID does not exist" containerID="0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623" Oct 31 03:25:24.328727 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.328668 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623"} err="failed to get container status \"0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623\": rpc error: code = NotFound desc = could not find container \"0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623\": container with ID starting with 0a8bf24850530e683c8946da4ad92f3579876852e856285241bad23851c25623 not found: ID does not exist" Oct 31 03:25:24.340035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.340006 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94"] Oct 31 03:25:24.343732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.343714 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-jhx94"] Oct 31 03:25:24.460567 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:24.460532 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" path="/var/lib/kubelet/pods/7ae6cf33-76ca-40b1-9424-59dadb8de118/volumes" Oct 31 03:25:38.985050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:38.985009 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7"] Oct 31 03:25:38.985678 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:38.985329 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" containerID="cri-o://981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d" gracePeriod=30 Oct 31 03:25:39.249966 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:39.249883 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:25:44.251335 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.251278 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:25:44.398494 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398467 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.398686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398507 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.398686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398540 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.398686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398596 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.398686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398620 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.398686 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398660 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398691 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398732 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398767 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token\") pod \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\" (UID: \"1f0c9ebd-eb44-42ed-b098-874f0511fde9\") " Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398800 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.398998 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399023 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:44.399061 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399042 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:44.399490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399356 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:44.399490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399371 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.399490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399410 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.399490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399423 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.399490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.399432 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.401305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.401282 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:44.401305 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.401293 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:44.401607 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.401591 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:44.401675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.401624 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw" pod "1f0c9ebd-eb44-42ed-b098-874f0511fde9" (UID: "1f0c9ebd-eb44-42ed-b098-874f0511fde9"). InnerVolumeSpecName "kube-api-access-xqvlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:44.455732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.455703 4854 generic.go:356] "Generic (PLEG): container finished" podID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerID="981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d" exitCode=0 Oct 31 03:25:44.455846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.455748 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" event={"ID":"1f0c9ebd-eb44-42ed-b098-874f0511fde9","Type":"ContainerDied","Data":"981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d"} Oct 31 03:25:44.455846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.455769 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" Oct 31 03:25:44.455846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.455786 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7" event={"ID":"1f0c9ebd-eb44-42ed-b098-874f0511fde9","Type":"ContainerDied","Data":"cbae21d8f6fcb5efd941445e2201ec94628cbcaee67ca62c42af68163ad97122"} Oct 31 03:25:44.455846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.455808 4854 scope.go:122] "RemoveContainer" containerID="981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d" Oct 31 03:25:44.466516 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.466454 4854 scope.go:122] "RemoveContainer" containerID="4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db" Oct 31 03:25:44.474837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.474752 4854 scope.go:122] "RemoveContainer" containerID="981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d" Oct 31 03:25:44.475052 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:44.475032 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d\": container with ID starting with 981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d not found: ID does not exist" containerID="981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d" Oct 31 03:25:44.475125 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.475060 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d"} err="failed to get container status \"981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d\": rpc error: code = NotFound desc = could not find container \"981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d\": container with ID starting with 981662fb9d211183e9952eccb03b84e042334a1e71ffe0914cee52b5a6aa8b1d not found: ID does not exist" Oct 31 03:25:44.475125 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.475077 4854 scope.go:122] "RemoveContainer" containerID="4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db" Oct 31 03:25:44.475400 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:25:44.475377 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db\": container with ID starting with 4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db not found: ID does not exist" containerID="4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db" Oct 31 03:25:44.475500 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.475405 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db"} err="failed to get container status \"4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db\": rpc error: code = NotFound desc = could not find container \"4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db\": container with ID starting with 4d8592721ea3fe8a92bcaf7dc9945171184b823fd4212d3c813c8cbfbcd8d4db not found: ID does not exist" Oct 31 03:25:44.482840 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.482815 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7"] Oct 31 03:25:44.488632 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.488615 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-0a531d-openshift-default-79bfd7b69-9f8x7"] Oct 31 03:25:44.500704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.500685 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqvlw\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-kube-api-access-xqvlw\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.500780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.500706 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.500780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.500719 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.500780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.500730 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:44.500780 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:44.500739 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0c9ebd-eb44-42ed-b098-874f0511fde9-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:46.456598 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:25:46.456570 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" path="/var/lib/kubelet/pods/1f0c9ebd-eb44-42ed-b098-874f0511fde9/volumes" Oct 31 03:26:05.873857 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:05.873816 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2"] Oct 31 03:26:05.874415 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:05.874116 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" podUID="f63bffd9-d5ca-4f28-8b98-42379e1df918" containerName="maas-api" containerID="cri-o://e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8" gracePeriod=30 Oct 31 03:26:06.126310 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.126240 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:26:06.239051 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.239022 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls\") pod \"f63bffd9-d5ca-4f28-8b98-42379e1df918\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " Oct 31 03:26:06.239242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.239072 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca\") pod \"f63bffd9-d5ca-4f28-8b98-42379e1df918\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " Oct 31 03:26:06.239242 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.239164 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6\" (UniqueName: \"kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6\") pod \"f63bffd9-d5ca-4f28-8b98-42379e1df918\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " Oct 31 03:26:06.239405 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.239242 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs\") pod \"f63bffd9-d5ca-4f28-8b98-42379e1df918\" (UID: \"f63bffd9-d5ca-4f28-8b98-42379e1df918\") " Oct 31 03:26:06.239564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.239541 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca" pod "f63bffd9-d5ca-4f28-8b98-42379e1df918" (UID: "f63bffd9-d5ca-4f28-8b98-42379e1df918"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:06.239695 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.239678 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63bffd9-d5ca-4f28-8b98-42379e1df918-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:06.241716 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.241692 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6" pod "f63bffd9-d5ca-4f28-8b98-42379e1df918" (UID: "f63bffd9-d5ca-4f28-8b98-42379e1df918"). InnerVolumeSpecName "kube-api-access-v87f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:06.241827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.241786 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls" pod "f63bffd9-d5ca-4f28-8b98-42379e1df918" (UID: "f63bffd9-d5ca-4f28-8b98-42379e1df918"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:06.243298 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.243273 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs" pod "f63bffd9-d5ca-4f28-8b98-42379e1df918" (UID: "f63bffd9-d5ca-4f28-8b98-42379e1df918"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:06.358362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.356175 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v87f6\" (UniqueName: \"kubernetes.io/projected/f63bffd9-d5ca-4f28-8b98-42379e1df918-kube-api-access-v87f6\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:06.358362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.356213 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:06.358362 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.356228 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63bffd9-d5ca-4f28-8b98-42379e1df918-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:06.572742 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.572702 4854 generic.go:356] "Generic (PLEG): container finished" podID="f63bffd9-d5ca-4f28-8b98-42379e1df918" containerID="e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8" exitCode=0 Oct 31 03:26:06.572921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.572750 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" Oct 31 03:26:06.572921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.572794 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" event={"ID":"f63bffd9-d5ca-4f28-8b98-42379e1df918","Type":"ContainerDied","Data":"e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8"} Oct 31 03:26:06.572921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.572864 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2" event={"ID":"f63bffd9-d5ca-4f28-8b98-42379e1df918","Type":"ContainerDied","Data":"7ec439cc7d7440da8dc8a2d689044af252cd02b79c343bc5caf5ae044127023e"} Oct 31 03:26:06.572921 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.572903 4854 scope.go:122] "RemoveContainer" containerID="e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8" Oct 31 03:26:06.582990 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.582974 4854 scope.go:122] "RemoveContainer" containerID="e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8" Oct 31 03:26:06.583317 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:06.583298 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8\": container with ID starting with e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8 not found: ID does not exist" containerID="e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8" Oct 31 03:26:06.583417 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.583329 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8"} err="failed to get container status \"e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8\": rpc error: code = NotFound desc = could not find container \"e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8\": container with ID starting with e3b7171afd1fd1f30e37344abd3e541eb46e9aa8eab694d72a71d4b28bbea5a8 not found: ID does not exist" Oct 31 03:26:06.590660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.590639 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2"] Oct 31 03:26:06.594941 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:06.594917 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebea4-76d9c685d4-plkz2"] Oct 31 03:26:08.455502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:08.455466 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63bffd9-d5ca-4f28-8b98-42379e1df918" path="/var/lib/kubelet/pods/f63bffd9-d5ca-4f28-8b98-42379e1df918/volumes" Oct 31 03:26:09.251057 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:09.251021 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:26:13.097779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:13.097742 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98"] Oct 31 03:26:13.099337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:13.098077 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" containerID="cri-o://5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255" gracePeriod=30 Oct 31 03:26:13.102097 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:13.102017 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm"] Oct 31 03:26:13.102263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:13.102227 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" containerID="cri-o://1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1" gracePeriod=30 Oct 31 03:26:17.501527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.501486 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64564ccd46-zhqr6"] Oct 31 03:26:17.502152 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502128 4854 cpu_manager.go:414] "containerMap: removing container" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502156 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502174 4854 cpu_manager.go:414] "containerMap: removing container" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502182 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502203 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502211 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502221 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502229 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" Oct 31 03:26:17.502238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502241 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502268 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502278 4854 cpu_manager.go:414] "containerMap: removing container" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502285 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502299 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502307 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502321 4854 cpu_manager.go:414] "containerMap: removing container" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502328 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502341 4854 cpu_manager.go:414] "containerMap: removing container" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502348 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502358 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502365 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502377 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502384 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502394 4854 cpu_manager.go:414] "containerMap: removing container" podUID="38943b33-ed6e-492b-beb8-0eb4bf8a8d81" containerName="payload-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502401 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38943b33-ed6e-492b-beb8-0eb4bf8a8d81" containerName="payload-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502417 4854 cpu_manager.go:414] "containerMap: removing container" podUID="37b2af3c-0bd3-4866-b0f0-69572d366e40" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502424 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b2af3c-0bd3-4866-b0f0-69572d366e40" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502437 4854 cpu_manager.go:414] "containerMap: removing container" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerName="payload-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502444 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerName="payload-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502454 4854 cpu_manager.go:414] "containerMap: removing container" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502462 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502479 4854 cpu_manager.go:414] "containerMap: removing container" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502486 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502500 4854 cpu_manager.go:414] "containerMap: removing container" podUID="521608fc-f682-4cf0-96c2-f529ae646f78" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502507 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521608fc-f682-4cf0-96c2-f529ae646f78" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502521 4854 cpu_manager.go:414] "containerMap: removing container" podUID="f63bffd9-d5ca-4f28-8b98-42379e1df918" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502527 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63bffd9-d5ca-4f28-8b98-42379e1df918" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502536 4854 cpu_manager.go:414] "containerMap: removing container" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502543 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502555 4854 cpu_manager.go:414] "containerMap: removing container" podUID="867896b1-871f-47e4-8a01-a283fe3e2810" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502561 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867896b1-871f-47e4-8a01-a283fe3e2810" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502574 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9bf4c889-9afb-4325-b343-ec7fb2251357" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502581 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf4c889-9afb-4325-b343-ec7fb2251357" containerName="maas-api" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502590 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502597 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502609 4854 cpu_manager.go:414] "containerMap: removing container" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502615 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502629 4854 cpu_manager.go:414] "containerMap: removing container" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502636 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502647 4854 cpu_manager.go:414] "containerMap: removing container" podUID="53edaf58-a577-4597-8c91-104af425b533" containerName="payload-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502654 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53edaf58-a577-4597-8c91-104af425b533" containerName="payload-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502665 4854 cpu_manager.go:414] "containerMap: removing container" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerName="payload-pre-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502673 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerName="payload-pre-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502686 4854 cpu_manager.go:414] "containerMap: removing container" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerName="discovery" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502694 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerName="discovery" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502705 4854 cpu_manager.go:414] "containerMap: removing container" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502712 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502725 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerName="payload-pre-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502733 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerName="payload-pre-processing" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502748 4854 cpu_manager.go:414] "containerMap: removing container" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" Oct 31 03:26:17.502721 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502755 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502769 4854 cpu_manager.go:414] "containerMap: removing container" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502777 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502789 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502797 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502806 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5eddc677-2ef2-412a-8dda-f2e80581ad59" containerName="payload-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502812 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eddc677-2ef2-412a-8dda-f2e80581ad59" containerName="payload-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502822 4854 cpu_manager.go:414] "containerMap: removing container" podUID="390be04b-2767-42a6-bed1-4466d0ca43f0" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502829 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390be04b-2767-42a6-bed1-4466d0ca43f0" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502841 4854 cpu_manager.go:414] "containerMap: removing container" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502848 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502865 4854 cpu_manager.go:414] "containerMap: removing container" podUID="0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502872 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502882 4854 cpu_manager.go:414] "containerMap: removing container" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502890 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.502993 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="521608fc-f682-4cf0-96c2-f529ae646f78" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503012 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63bffd9-d5ca-4f28-8b98-42379e1df918" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503022 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c17918e-87b0-4d8c-88e5-d072a50b9f59" containerName="main" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503031 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="867896b1-871f-47e4-8a01-a283fe3e2810" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503039 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503048 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf4c889-9afb-4325-b343-ec7fb2251357" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503058 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ecbffa-bea3-48b4-aa3d-5ab7e14603a0" containerName="main" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503069 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503078 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae6cf33-76ca-40b1-9424-59dadb8de118" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503086 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="07765218-86e7-419d-8ae8-a981d9b47062" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503096 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="53edaf58-a577-4597-8c91-104af425b533" containerName="payload-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503105 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4bede3-2df5-4bc8-bda5-edcf1f5ef0bc" containerName="payload-pre-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503113 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e17e36c-cf1f-485f-a02f-a2c57b1f699f" containerName="discovery" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503121 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb6194f-74ea-4818-bea2-fea075d97173" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503130 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a847ac29-5362-45ac-9715-77a1c1a90e2b" containerName="payload-pre-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503139 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="3597b530-55e8-43f0-ae5f-bc83519f7137" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503148 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503157 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a463dafe-3ebd-4c43-9376-934853ff4b36" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503165 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eddc677-2ef2-412a-8dda-f2e80581ad59" containerName="payload-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503174 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="390be04b-2767-42a6-bed1-4466d0ca43f0" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503181 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d33873-bc9b-4ccc-bee4-7e50adf8e730" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503192 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503200 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef60be2-e0aa-4791-90e6-e14cdf3dbb5d" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503210 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0c9ebd-eb44-42ed-b098-874f0511fde9" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503221 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503230 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="492d17b7-7dca-48ca-991e-62e32e839673" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503239 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503267 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503277 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503285 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ce5a45-f5ec-4a49-a765-440318032ee5" containerName="main" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503296 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503306 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="42347c06-7df7-4f5c-acd3-9127f8f42302" containerName="main" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503315 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a112cb-cb85-47b4-b21c-dcef4e92ac61" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503323 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba893f3-b888-41af-a479-93b4965a7813" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503331 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="38943b33-ed6e-492b-beb8-0eb4bf8a8d81" containerName="payload-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503340 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b2af3c-0bd3-4866-b0f0-69572d366e40" containerName="maas-api" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503350 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503359 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="743857fb-f1c4-4cc7-a6c3-ba01a0e070cf" containerName="payload-processing" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503368 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="327587c2-603d-4c5e-8c05-f06f49e634f9" containerName="main" Oct 31 03:26:17.505627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.503377 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54ac6d7-4e3c-4438-adaf-f0998762ad67" containerName="istio-proxy" Oct 31 03:26:17.508650 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.507216 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.510741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.510715 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 03:26:17.510741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.510732 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 03:26:17.511011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.510796 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pkwql\"" Oct 31 03:26:17.511011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.510875 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 03:26:17.526472 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.524339 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64564ccd46-zhqr6"] Oct 31 03:26:17.577386 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.577354 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.577545 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.577400 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.577620 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.577538 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjml\" (UniqueName: \"kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.678216 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.678180 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjml\" (UniqueName: \"kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.678439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.678348 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.678439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.678381 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.681849 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.681819 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.681991 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.681867 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.690355 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.690331 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjml\" (UniqueName: \"kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml\") pod \"maas-controller-64564ccd46-zhqr6\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:17.843706 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:17.843622 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:18.452294 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.450275 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64564ccd46-zhqr6"] Oct 31 03:26:18.551978 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:26:18.551919 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd73ec83_8f93_423c_9bae_7b8a1f5f1608.slice/crio-48143b67ec840d75e8b116b8d0c897b8a69ff7d24f0101fb4b06d9f293b70dc3 WatchSource:0}: Error finding container 48143b67ec840d75e8b116b8d0c897b8a69ff7d24f0101fb4b06d9f293b70dc3: Status 404 returned error can't find the container with id 48143b67ec840d75e8b116b8d0c897b8a69ff7d24f0101fb4b06d9f293b70dc3 Oct 31 03:26:18.562869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.553960 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:26:18.680239 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.680134 4854 generic.go:356] "Generic (PLEG): container finished" podID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerID="5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255" exitCode=0 Oct 31 03:26:18.680487 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.680294 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" event={"ID":"9b2d1aac-33ad-47e9-a872-6b33e04b386e","Type":"ContainerDied","Data":"5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255"} Oct 31 03:26:18.702892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.702857 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-zhqr6" event={"ID":"fd73ec83-8f93-423c-9bae-7b8a1f5f1608","Type":"ContainerStarted","Data":"48143b67ec840d75e8b116b8d0c897b8a69ff7d24f0101fb4b06d9f293b70dc3"} Oct 31 03:26:18.714673 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.714590 4854 generic.go:356] "Generic (PLEG): container finished" podID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerID="1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1" exitCode=0 Oct 31 03:26:18.714907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:18.714647 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" event={"ID":"e5cac45d-e413-4766-84e7-50b4ec39edde","Type":"ContainerDied","Data":"1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1"} Oct 31 03:26:19.572660 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.572631 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:26:19.623217 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.613528 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694583 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694639 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694677 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694707 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694768 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694831 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694856 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694886 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.699623 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.694937 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert\") pod \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\" (UID: \"9b2d1aac-33ad-47e9-a872-6b33e04b386e\") " Oct 31 03:26:19.701822 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.701447 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:19.703886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.703340 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.703886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.703635 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.703886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.703857 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.704383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.704349 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.744565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.729744 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "kube-api-access-8r5lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:19.779004 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.777384 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:26:19.780969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.780908 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:19.782556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.781373 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy" pod "9b2d1aac-33ad-47e9-a872-6b33e04b386e" (UID: "9b2d1aac-33ad-47e9-a872-6b33e04b386e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.782556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.781782 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" event={"ID":"e5cac45d-e413-4766-84e7-50b4ec39edde","Type":"ContainerDied","Data":"a991ef504f7a9322b72411f68d2bc3a17c5c3be15f66a2e65d15d87450921733"} Oct 31 03:26:19.782556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.781859 4854 scope.go:122] "RemoveContainer" containerID="1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1" Oct 31 03:26:19.782556 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.782121 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801527 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801585 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801633 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801702 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801739 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801808 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801853 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801940 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.801975 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs\") pod \"e5cac45d-e413-4766-84e7-50b4ec39edde\" (UID: \"e5cac45d-e413-4766-84e7-50b4ec39edde\") " Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802319 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802338 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802351 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802365 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802379 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802392 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5lr\" (UniqueName: \"kubernetes.io/projected/9b2d1aac-33ad-47e9-a872-6b33e04b386e-kube-api-access-8r5lr\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802405 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802417 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802430 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2d1aac-33ad-47e9-a872-6b33e04b386e-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.802905 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:19.803691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.803392 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.822271 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.814534 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.830505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.829348 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.830505 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.829968 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.850269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.843621 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" event={"ID":"9b2d1aac-33ad-47e9-a872-6b33e04b386e","Type":"ContainerDied","Data":"3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da"} Oct 31 03:26:19.850269 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.843851 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98" Oct 31 03:26:19.850689 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.850637 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:26:19.875957 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.875427 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "kube-api-access-4rk4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:19.877009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.876938 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:19.878322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.878155 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy" pod "e5cac45d-e413-4766-84e7-50b4ec39edde" (UID: "e5cac45d-e413-4766-84e7-50b4ec39edde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.903928 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.903983 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904001 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904016 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904030 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904045 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cac45d-e413-4766-84e7-50b4ec39edde-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904060 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904075 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rk4r\" (UniqueName: \"kubernetes.io/projected/e5cac45d-e413-4766-84e7-50b4ec39edde-kube-api-access-4rk4r\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.905901 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.904092 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cac45d-e413-4766-84e7-50b4ec39edde-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.930367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.927407 4854 scope.go:122] "RemoveContainer" containerID="5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255" Oct 31 03:26:19.930367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.929616 4854 scope.go:122] "RemoveContainer" containerID="5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255" Oct 31 03:26:19.943337 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.943276 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98"] Oct 31 03:26:19.945473 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.945447 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98"] Oct 31 03:26:19.958733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.958509 4854 scope.go:122] "RemoveContainer" containerID="1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1" Oct 31 03:26:19.958733 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:19.958663 4854 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98_openshift-ingress_9b2d1aac-33ad-47e9-a872-6b33e04b386e_0 in pod sandbox 3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da from index: no such id: '5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255'" containerID="5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255" Oct 31 03:26:19.958733 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:19.958702 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-pzt98_openshift-ingress_9b2d1aac-33ad-47e9-a872-6b33e04b386e_0 in pod sandbox 3c72ae5d8a0728f2c1b3b3ff1c8135ae1a1b11b59c5d05d215e8a19ba20ef5da from index: no such id: '5f191b88713f6b025cdf122e24beba3e216573ff944eca330dca3e2d93b3c255'" Oct 31 03:26:19.959397 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:19.959234 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1\": container with ID starting with 1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1 not found: ID does not exist" containerID="1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1" Oct 31 03:26:19.959397 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:19.959296 4854 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1\": rpc error: code = NotFound desc = could not find container \"1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1\": container with ID starting with 1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1 not found: ID does not exist" containerID="1d2a5f5e951c1ef04e57f8e4570db1ebb2bcc671990029b2a45d1c9ce2f6e3a1" Oct 31 03:26:20.102994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:20.102960 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm"] Oct 31 03:26:20.106834 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:20.106807 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-prkkm"] Oct 31 03:26:20.456827 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:20.456787 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" path="/var/lib/kubelet/pods/9b2d1aac-33ad-47e9-a872-6b33e04b386e/volumes" Oct 31 03:26:20.457366 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:20.457345 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" path="/var/lib/kubelet/pods/e5cac45d-e413-4766-84e7-50b4ec39edde/volumes" Oct 31 03:26:21.857832 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:21.857785 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-zhqr6" event={"ID":"fd73ec83-8f93-423c-9bae-7b8a1f5f1608","Type":"ContainerStarted","Data":"495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c"} Oct 31 03:26:21.858222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:21.857994 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:21.877601 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:21.877548 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64564ccd46-zhqr6" podStartSLOduration=2.468493942 podStartE2EDuration="4.877536422s" totalImagesPullingTime="2.40904248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:21.875438372 +0000 UTC m=+1865.572605680" watchObservedRunningTime="2026-10-31 03:26:21.877536422 +0000 UTC m=+1865.574703726" Oct 31 03:26:32.864831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:32.864793 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:39.250238 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:39.250199 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:26:51.294880 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.294841 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64564ccd46-zhqr6"] Oct 31 03:26:51.300761 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.295047 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64564ccd46-zhqr6" podUID="fd73ec83-8f93-423c-9bae-7b8a1f5f1608" containerName="manager" containerID="cri-o://495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c" gracePeriod=30 Oct 31 03:26:51.651787 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.651765 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:51.786065 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.786033 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert\") pod \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " Oct 31 03:26:51.786279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.786125 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml\" (UniqueName: \"kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml\") pod \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " Oct 31 03:26:51.786279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.786176 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs\") pod \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\" (UID: \"fd73ec83-8f93-423c-9bae-7b8a1f5f1608\") " Oct 31 03:26:51.788380 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.788349 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml" pod "fd73ec83-8f93-423c-9bae-7b8a1f5f1608" (UID: "fd73ec83-8f93-423c-9bae-7b8a1f5f1608"). InnerVolumeSpecName "kube-api-access-szjml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:51.788489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.788391 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs" pod "fd73ec83-8f93-423c-9bae-7b8a1f5f1608" (UID: "fd73ec83-8f93-423c-9bae-7b8a1f5f1608"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:51.788489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.788403 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert" pod "fd73ec83-8f93-423c-9bae-7b8a1f5f1608" (UID: "fd73ec83-8f93-423c-9bae-7b8a1f5f1608"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:51.886884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.886829 4854 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-webhook-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:51.886884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.886853 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szjml\" (UniqueName: \"kubernetes.io/projected/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-kube-api-access-szjml\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:51.886884 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:51.886863 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd73ec83-8f93-423c-9bae-7b8a1f5f1608-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:52.002631 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.002600 4854 generic.go:356] "Generic (PLEG): container finished" podID="fd73ec83-8f93-423c-9bae-7b8a1f5f1608" containerID="495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c" exitCode=0 Oct 31 03:26:52.002757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.002653 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-zhqr6" Oct 31 03:26:52.002757 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.002688 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-zhqr6" event={"ID":"fd73ec83-8f93-423c-9bae-7b8a1f5f1608","Type":"ContainerDied","Data":"495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c"} Oct 31 03:26:52.002896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.002779 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-zhqr6" event={"ID":"fd73ec83-8f93-423c-9bae-7b8a1f5f1608","Type":"ContainerDied","Data":"48143b67ec840d75e8b116b8d0c897b8a69ff7d24f0101fb4b06d9f293b70dc3"} Oct 31 03:26:52.002896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.002811 4854 scope.go:122] "RemoveContainer" containerID="495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c" Oct 31 03:26:52.014600 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.014576 4854 scope.go:122] "RemoveContainer" containerID="495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c" Oct 31 03:26:52.014995 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:26:52.014968 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c\": container with ID starting with 495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c not found: ID does not exist" containerID="495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c" Oct 31 03:26:52.015102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.015004 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c"} err="failed to get container status \"495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c\": rpc error: code = NotFound desc = could not find container \"495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c\": container with ID starting with 495c056921ee0d8d08e80aa2f1ad57f23426aeffc8f6930b0e21bd73dd33cd3c not found: ID does not exist" Oct 31 03:26:52.024477 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.024456 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64564ccd46-zhqr6"] Oct 31 03:26:52.028751 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.028731 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64564ccd46-zhqr6"] Oct 31 03:26:52.455560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:52.455532 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd73ec83-8f93-423c-9bae-7b8a1f5f1608" path="/var/lib/kubelet/pods/fd73ec83-8f93-423c-9bae-7b8a1f5f1608/volumes" Oct 31 03:26:53.261385 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.261350 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64564ccd46-thk28"] Oct 31 03:26:53.262010 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.261989 4854 cpu_manager.go:414] "containerMap: removing container" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" Oct 31 03:26:53.262073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262012 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" Oct 31 03:26:53.262073 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262030 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262072 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc26c32-e947-4d59-9238-abd4cf8d261c" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262091 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262099 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93ff988-5bc5-495c-99fa-2f8bdbe432b6" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262117 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262125 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f386509-0bc2-4565-9b54-f4efa32fad2c" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262141 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" Oct 31 03:26:53.262156 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262148 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262167 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262174 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262208 4854 cpu_manager.go:414] "containerMap: removing container" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262216 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262228 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fd73ec83-8f93-423c-9bae-7b8a1f5f1608" containerName="manager" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262235 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd73ec83-8f93-423c-9bae-7b8a1f5f1608" containerName="manager" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262372 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cac45d-e413-4766-84e7-50b4ec39edde" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262390 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ebd920-0c0a-4cee-b2ef-8b307ce014f3" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262402 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2d1aac-33ad-47e9-a872-6b33e04b386e" containerName="istio-proxy" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262410 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd73ec83-8f93-423c-9bae-7b8a1f5f1608" containerName="manager" Oct 31 03:26:53.262488 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.262422 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="9661d7af-7aba-4d48-bf9a-1f97f804b29a" containerName="istio-proxy" Oct 31 03:26:53.268509 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.268489 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.275599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.275500 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 03:26:53.276020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.275995 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pkwql\"" Oct 31 03:26:53.276139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.276092 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 03:26:53.276483 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.276411 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 03:26:53.278192 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.278168 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64564ccd46-thk28"] Oct 31 03:26:53.401095 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.401062 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff88014-ac5d-4d2d-97a6-6383a4a619af-webhook-cert\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.401279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.401105 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff88014-ac5d-4d2d-97a6-6383a4a619af-metrics-certs\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.401279 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.401183 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xghl\" (UniqueName: \"kubernetes.io/projected/cff88014-ac5d-4d2d-97a6-6383a4a619af-kube-api-access-8xghl\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.502521 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.502490 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff88014-ac5d-4d2d-97a6-6383a4a619af-webhook-cert\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.502927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.502531 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff88014-ac5d-4d2d-97a6-6383a4a619af-metrics-certs\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.502927 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.502675 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xghl\" (UniqueName: \"kubernetes.io/projected/cff88014-ac5d-4d2d-97a6-6383a4a619af-kube-api-access-8xghl\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.505161 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.505135 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff88014-ac5d-4d2d-97a6-6383a4a619af-webhook-cert\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.505322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.505302 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff88014-ac5d-4d2d-97a6-6383a4a619af-metrics-certs\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.514503 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.514451 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xghl\" (UniqueName: \"kubernetes.io/projected/cff88014-ac5d-4d2d-97a6-6383a4a619af-kube-api-access-8xghl\") pod \"maas-controller-64564ccd46-thk28\" (UID: \"cff88014-ac5d-4d2d-97a6-6383a4a619af\") " pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.585367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.585348 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:53.863314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:53.863283 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64564ccd46-thk28"] Oct 31 03:26:53.865492 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:26:53.865462 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff88014_ac5d_4d2d_97a6_6383a4a619af.slice/crio-bd8f798b6246956660a948b2999f7b716b9f0dc22882edc55629798ded9bafee WatchSource:0}: Error finding container bd8f798b6246956660a948b2999f7b716b9f0dc22882edc55629798ded9bafee: Status 404 returned error can't find the container with id bd8f798b6246956660a948b2999f7b716b9f0dc22882edc55629798ded9bafee Oct 31 03:26:54.015054 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:54.015024 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-thk28" event={"ID":"cff88014-ac5d-4d2d-97a6-6383a4a619af","Type":"ContainerStarted","Data":"bd8f798b6246956660a948b2999f7b716b9f0dc22882edc55629798ded9bafee"} Oct 31 03:26:55.029657 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:55.029616 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64564ccd46-thk28" event={"ID":"cff88014-ac5d-4d2d-97a6-6383a4a619af","Type":"ContainerStarted","Data":"7d773bade73fa22345d1196a611347feae6be5636ec9737620e00d0871fb8154"} Oct 31 03:26:55.030146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:55.029761 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:26:55.050159 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:26:55.050113 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64564ccd46-thk28" podStartSLOduration=1.602381216 podStartE2EDuration="2.050100388s" totalImagesPullingTime="447.719172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:55.048291113 +0000 UTC m=+1898.745458418" watchObservedRunningTime="2026-10-31 03:26:55.050100388 +0000 UTC m=+1898.747267692" Oct 31 03:27:06.037131 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:27:06.037098 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64564ccd46-thk28" Oct 31 03:27:09.249705 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:27:09.249668 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:27:39.249555 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:27:39.249471 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:28:09.250012 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:28:09.249977 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:28:39.250645 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:28:39.250608 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:29:09.251278 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:29:09.251208 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:29:39.251186 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:29:39.251150 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:30:09.250696 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:30:09.250659 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:30:19.717727 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:30:19.717696 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:30:19.727050 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:30:19.727029 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:30:39.250047 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:30:39.249968 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:31:09.250085 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:31:09.250043 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:31:39.250878 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:31:39.250837 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:31:43.989994 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:43.989961 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl"] Oct 31 03:31:43.993692 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:43.993672 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.001984 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.001957 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-01b345\"" Oct 31 03:31:44.004427 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.004390 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl"] Oct 31 03:31:44.062893 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.062861 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg"] Oct 31 03:31:44.066528 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.066506 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.074412 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.074388 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ce5f680e-dockercfg-qp6xl\"" Oct 31 03:31:44.077263 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.077224 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg"] Oct 31 03:31:44.128935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.128905 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.129075 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.128975 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfs4\" (UniqueName: \"kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4\") pod \"payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.229998 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.229965 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.230177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.230009 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfs4\" (UniqueName: \"kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4\") pod \"payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.230177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.230134 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.230177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.230167 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpj9\" (UniqueName: \"kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9\") pod \"payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.230909 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.230882 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.241520 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.241468 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfs4\" (UniqueName: \"kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4\") pod \"payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.308587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.308563 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.331707 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.331683 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpj9\" (UniqueName: \"kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9\") pod \"payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.331828 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.331737 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.332609 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.332585 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.343670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.343640 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpj9\" (UniqueName: \"kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9\") pod \"payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.380670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.380590 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.585023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.584986 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl"] Oct 31 03:31:44.587679 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:31:44.587640 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode267b6ee_500c_4a14_9d8a_3c3a84a09a36.slice/crio-745282213c57db0415738e7a9114d511fc5da1323c7092ffad6723b752be0df8 WatchSource:0}: Error finding container 745282213c57db0415738e7a9114d511fc5da1323c7092ffad6723b752be0df8: Status 404 returned error can't find the container with id 745282213c57db0415738e7a9114d511fc5da1323c7092ffad6723b752be0df8 Oct 31 03:31:44.664121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.664097 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg"] Oct 31 03:31:44.666514 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:31:44.666490 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce003bb_b37c_4b51_9cbc_1ae38639bf9e.slice/crio-95605a946bd27f67d6f30886be5b1003f0d935e65b58a21af2c8b306e0ca6796 WatchSource:0}: Error finding container 95605a946bd27f67d6f30886be5b1003f0d935e65b58a21af2c8b306e0ca6796: Status 404 returned error can't find the container with id 95605a946bd27f67d6f30886be5b1003f0d935e65b58a21af2c8b306e0ca6796 Oct 31 03:31:44.959873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.959785 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" event={"ID":"e267b6ee-500c-4a14-9d8a-3c3a84a09a36","Type":"ContainerStarted","Data":"fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03"} Oct 31 03:31:44.959873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.959844 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" event={"ID":"e267b6ee-500c-4a14-9d8a-3c3a84a09a36","Type":"ContainerStarted","Data":"745282213c57db0415738e7a9114d511fc5da1323c7092ffad6723b752be0df8"} Oct 31 03:31:44.960106 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.959882 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:44.961394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.961362 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" event={"ID":"fce003bb-b37c-4b51-9cbc-1ae38639bf9e","Type":"ContainerStarted","Data":"75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1"} Oct 31 03:31:44.961394 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.961395 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" event={"ID":"fce003bb-b37c-4b51-9cbc-1ae38639bf9e","Type":"ContainerStarted","Data":"95605a946bd27f67d6f30886be5b1003f0d935e65b58a21af2c8b306e0ca6796"} Oct 31 03:31:44.961596 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.961491 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:31:44.979675 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.979636 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" podStartSLOduration=1.979625323 podStartE2EDuration="1.979625323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:44.978319581 +0000 UTC m=+2188.675486877" watchObservedRunningTime="2026-10-31 03:31:44.979625323 +0000 UTC m=+2188.676792626" Oct 31 03:31:44.996568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:44.996460 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" podStartSLOduration=0.996444377 podStartE2EDuration="996.444377ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:44.995384909 +0000 UTC m=+2188.692552215" watchObservedRunningTime="2026-10-31 03:31:44.996444377 +0000 UTC m=+2188.693611877" Oct 31 03:31:55.969443 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:55.969407 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:31:55.971993 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:31:55.969484 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:32:09.251027 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:32:09.250984 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:32:39.251268 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:32:39.251214 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:32:51.613504 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:32:51.613466 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg"] Oct 31 03:32:51.623119 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:32:51.623082 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl"] Oct 31 03:32:56.614576 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:32:56.614538 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerName="payload-processing" containerID="cri-o://75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1" gracePeriod=25 Oct 31 03:32:56.624016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:32:56.623981 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerName="payload-pre-processing" containerID="cri-o://fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03" gracePeriod=25 Oct 31 03:33:05.967949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:05.967908 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:33:05.967949 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:05.967935 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:33:09.250707 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:33:09.250669 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:33:15.967862 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:15.967823 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:33:15.968325 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:15.967869 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:33:16.875288 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:16.875210 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:33:16.936139 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:16.936113 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:33:16.971907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:16.971875 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume\") pod \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " Oct 31 03:33:16.972320 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:16.971958 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9\" (UniqueName: \"kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9\") pod \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\" (UID: \"fce003bb-b37c-4b51-9cbc-1ae38639bf9e\") " Oct 31 03:33:16.972379 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:16.972333 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume" pod "fce003bb-b37c-4b51-9cbc-1ae38639bf9e" (UID: "fce003bb-b37c-4b51-9cbc-1ae38639bf9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:16.974160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:16.974139 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9" pod "fce003bb-b37c-4b51-9cbc-1ae38639bf9e" (UID: "fce003bb-b37c-4b51-9cbc-1ae38639bf9e"). InnerVolumeSpecName "kube-api-access-zdpj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:17.072741 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.072712 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume\") pod \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " Oct 31 03:33:17.072896 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.072771 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4\" (UniqueName: \"kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4\") pod \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\" (UID: \"e267b6ee-500c-4a14-9d8a-3c3a84a09a36\") " Oct 31 03:33:17.073121 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.073102 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume" pod "e267b6ee-500c-4a14-9d8a-3c3a84a09a36" (UID: "e267b6ee-500c-4a14-9d8a-3c3a84a09a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:17.073175 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.073165 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:17.073211 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.073182 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpj9\" (UniqueName: \"kubernetes.io/projected/fce003bb-b37c-4b51-9cbc-1ae38639bf9e-kube-api-access-zdpj9\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:17.075088 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.075071 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4" pod "e267b6ee-500c-4a14-9d8a-3c3a84a09a36" (UID: "e267b6ee-500c-4a14-9d8a-3c3a84a09a36"). InnerVolumeSpecName "kube-api-access-jsfs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:17.174489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.174421 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsfs4\" (UniqueName: \"kubernetes.io/projected/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-kube-api-access-jsfs4\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:17.174489 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.174444 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e267b6ee-500c-4a14-9d8a-3c3a84a09a36-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:17.446979 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.446893 4854 generic.go:356] "Generic (PLEG): container finished" podID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerID="75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1" exitCode=0 Oct 31 03:33:17.446979 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.446941 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" Oct 31 03:33:17.447198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.446976 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" event={"ID":"fce003bb-b37c-4b51-9cbc-1ae38639bf9e","Type":"ContainerDied","Data":"75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1"} Oct 31 03:33:17.447198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.447031 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg" event={"ID":"fce003bb-b37c-4b51-9cbc-1ae38639bf9e","Type":"ContainerDied","Data":"95605a946bd27f67d6f30886be5b1003f0d935e65b58a21af2c8b306e0ca6796"} Oct 31 03:33:17.447198 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.447059 4854 scope.go:122] "RemoveContainer" containerID="75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1" Oct 31 03:33:17.454873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.454845 4854 generic.go:356] "Generic (PLEG): container finished" podID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerID="fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03" exitCode=0 Oct 31 03:33:17.455102 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.455080 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" Oct 31 03:33:17.455315 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.455083 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" event={"ID":"e267b6ee-500c-4a14-9d8a-3c3a84a09a36","Type":"ContainerDied","Data":"fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03"} Oct 31 03:33:17.455480 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.455450 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl" event={"ID":"e267b6ee-500c-4a14-9d8a-3c3a84a09a36","Type":"ContainerDied","Data":"745282213c57db0415738e7a9114d511fc5da1323c7092ffad6723b752be0df8"} Oct 31 03:33:17.464551 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.464536 4854 scope.go:122] "RemoveContainer" containerID="75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1" Oct 31 03:33:17.464880 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:33:17.464859 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1\": container with ID starting with 75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1 not found: ID does not exist" containerID="75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1" Oct 31 03:33:17.464977 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.464885 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1"} err="failed to get container status \"75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1\": rpc error: code = NotFound desc = could not find container \"75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1\": container with ID starting with 75c61806a2d6f7e54e4b4ff4b94acd9252fbfa0e62e4831cf6356bebb75ceaf1 not found: ID does not exist" Oct 31 03:33:17.464977 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.464902 4854 scope.go:122] "RemoveContainer" containerID="fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03" Oct 31 03:33:17.470058 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.470040 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg"] Oct 31 03:33:17.473989 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.473967 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-01b345-59788cbf4kqrqg"] Oct 31 03:33:17.474283 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.474261 4854 scope.go:122] "RemoveContainer" containerID="fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03" Oct 31 03:33:17.474562 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:33:17.474533 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03\": container with ID starting with fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03 not found: ID does not exist" containerID="fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03" Oct 31 03:33:17.474661 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.474565 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03"} err="failed to get container status \"fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03\": rpc error: code = NotFound desc = could not find container \"fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03\": container with ID starting with fb9b29be07fb35e97e6229ac2681a3381b3402df5bb54d8bc21feb5f53c6fb03 not found: ID does not exist" Oct 31 03:33:17.483041 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.483010 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl"] Oct 31 03:33:17.487154 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:17.487132 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-01b345-65b6f8kdkl"] Oct 31 03:33:18.455364 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:18.455325 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" path="/var/lib/kubelet/pods/e267b6ee-500c-4a14-9d8a-3c3a84a09a36/volumes" Oct 31 03:33:18.455763 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:33:18.455703 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" path="/var/lib/kubelet/pods/fce003bb-b37c-4b51-9cbc-1ae38639bf9e/volumes" Oct 31 03:33:39.249620 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:33:39.249537 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:34:09.251614 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:34:09.251576 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:34:39.250276 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:34:39.250226 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:35:09.251188 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:35:09.251156 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:35:19.807814 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:19.807786 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:35:19.815436 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:19.815413 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:35:22.387885 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.387840 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm"] Oct 31 03:35:22.388317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.388301 4854 cpu_manager.go:414] "containerMap: removing container" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerName="payload-pre-processing" Oct 31 03:35:22.388367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.388320 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerName="payload-pre-processing" Oct 31 03:35:22.388367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.388340 4854 cpu_manager.go:414] "containerMap: removing container" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerName="payload-processing" Oct 31 03:35:22.388367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.388345 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerName="payload-processing" Oct 31 03:35:22.388466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.388424 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce003bb-b37c-4b51-9cbc-1ae38639bf9e" containerName="payload-processing" Oct 31 03:35:22.388466 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.388440 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="e267b6ee-500c-4a14-9d8a-3c3a84a09a36" containerName="payload-pre-processing" Oct 31 03:35:22.391747 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.391730 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.395368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.395343 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d7ca5070-dockercfg-jxqpr\"" Oct 31 03:35:22.395804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.395785 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-31e475\"" Oct 31 03:35:22.408374 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.408354 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm"] Oct 31 03:35:22.448224 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.448196 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx"] Oct 31 03:35:22.451863 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.451842 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.466434 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.466408 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx"] Oct 31 03:35:22.562811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.562775 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6b4l\" (UniqueName: \"kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l\") pod \"payload-processing-e2e-worker-single-main-31e475-696b45845tpplx\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.563035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.562821 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-31e475-696b45845tpplx\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.563035 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.562982 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zbsg\" (UniqueName: \"kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg\") pod \"payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.563166 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.563075 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.664620 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.664538 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.664620 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.664608 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6b4l\" (UniqueName: \"kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l\") pod \"payload-processing-e2e-worker-single-main-31e475-696b45845tpplx\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.664872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.664632 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-31e475-696b45845tpplx\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.664872 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.664672 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zbsg\" (UniqueName: \"kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg\") pod \"payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.665539 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.665511 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-31e475-696b45845tpplx\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.665652 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.665512 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.676196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.676173 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zbsg\" (UniqueName: \"kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg\") pod \"payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.676557 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.676534 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6b4l\" (UniqueName: \"kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l\") pod \"payload-processing-e2e-worker-single-main-31e475-696b45845tpplx\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.704322 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.704300 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:22.764683 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.764649 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:22.983602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:22.983574 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm"] Oct 31 03:35:22.985371 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:35:22.985330 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e90998_0ef9_42a6_aa06_e600ff01e97e.slice/crio-d3d98318cfe5592b1a33921d2639d76a8f4a4fe4362ce96bc0962d96d804d5de WatchSource:0}: Error finding container d3d98318cfe5592b1a33921d2639d76a8f4a4fe4362ce96bc0962d96d804d5de: Status 404 returned error can't find the container with id d3d98318cfe5592b1a33921d2639d76a8f4a4fe4362ce96bc0962d96d804d5de Oct 31 03:35:23.062752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:23.062720 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx"] Oct 31 03:35:23.065029 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:35:23.065005 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946874be_d475_41a5_a53e_78a44039a0ad.slice/crio-249723b31a19d51c16f5d51cf7fcb965f0c26e46235c43b9ff30c21bd314ece7 WatchSource:0}: Error finding container 249723b31a19d51c16f5d51cf7fcb965f0c26e46235c43b9ff30c21bd314ece7: Status 404 returned error can't find the container with id 249723b31a19d51c16f5d51cf7fcb965f0c26e46235c43b9ff30c21bd314ece7 Oct 31 03:35:23.086731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:23.086701 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" event={"ID":"946874be-d475-41a5-a53e-78a44039a0ad","Type":"ContainerStarted","Data":"249723b31a19d51c16f5d51cf7fcb965f0c26e46235c43b9ff30c21bd314ece7"} Oct 31 03:35:23.088349 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:23.088321 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" event={"ID":"a2e90998-0ef9-42a6-aa06-e600ff01e97e","Type":"ContainerStarted","Data":"614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f"} Oct 31 03:35:23.088437 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:23.088370 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" event={"ID":"a2e90998-0ef9-42a6-aa06-e600ff01e97e","Type":"ContainerStarted","Data":"d3d98318cfe5592b1a33921d2639d76a8f4a4fe4362ce96bc0962d96d804d5de"} Oct 31 03:35:23.088478 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:23.088431 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:23.113755 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:23.113694 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" podStartSLOduration=1.11367983 podStartE2EDuration="1.11367983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:23.112709107 +0000 UTC m=+2406.809876413" watchObservedRunningTime="2026-10-31 03:35:23.11367983 +0000 UTC m=+2406.810847134" Oct 31 03:35:24.094879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:24.094842 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" event={"ID":"946874be-d475-41a5-a53e-78a44039a0ad","Type":"ContainerStarted","Data":"6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db"} Oct 31 03:35:24.095307 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:24.094945 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:24.111719 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:24.111674 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" podStartSLOduration=2.111663557 podStartE2EDuration="2.111663557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:24.111542454 +0000 UTC m=+2407.808709762" watchObservedRunningTime="2026-10-31 03:35:24.111663557 +0000 UTC m=+2407.808830863" Oct 31 03:35:34.098194 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:34.098166 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:35:35.104982 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:35:35.104951 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:35:39.250376 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:35:39.250339 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:36:09.250209 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:36:09.250165 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:36:29.612018 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:29.611941 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx"] Oct 31 03:36:29.623899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:29.623855 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm"] Oct 31 03:36:34.612533 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:34.612489 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" podUID="946874be-d475-41a5-a53e-78a44039a0ad" containerName="payload-processing" containerID="cri-o://6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db" gracePeriod=25 Oct 31 03:36:34.624587 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:34.624559 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerName="payload-pre-processing" containerID="cri-o://614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f" gracePeriod=25 Oct 31 03:36:35.103470 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:35.103438 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" podUID="946874be-d475-41a5-a53e-78a44039a0ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:36:39.250072 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:36:39.250035 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:36:44.096383 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:44.096343 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:36:45.103234 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:45.103191 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" podUID="946874be-d475-41a5-a53e-78a44039a0ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:36:47.303275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:47.299351 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:36:47.303275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:47.299423 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:54.096929 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:54.096885 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:36:54.885062 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:54.885007 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:36:54.916820 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:54.916793 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:36:55.011047 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.011022 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l\" (UniqueName: \"kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l\") pod \"946874be-d475-41a5-a53e-78a44039a0ad\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " Oct 31 03:36:55.011230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.011063 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume\") pod \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " Oct 31 03:36:55.011230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.011126 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg\" (UniqueName: \"kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg\") pod \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\" (UID: \"a2e90998-0ef9-42a6-aa06-e600ff01e97e\") " Oct 31 03:36:55.011230 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.011174 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume\") pod \"946874be-d475-41a5-a53e-78a44039a0ad\" (UID: \"946874be-d475-41a5-a53e-78a44039a0ad\") " Oct 31 03:36:55.011527 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.011498 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume" pod "a2e90998-0ef9-42a6-aa06-e600ff01e97e" (UID: "a2e90998-0ef9-42a6-aa06-e600ff01e97e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:55.011646 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.011542 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume" pod "946874be-d475-41a5-a53e-78a44039a0ad" (UID: "946874be-d475-41a5-a53e-78a44039a0ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:55.013518 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.013499 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg" pod "a2e90998-0ef9-42a6-aa06-e600ff01e97e" (UID: "a2e90998-0ef9-42a6-aa06-e600ff01e97e"). InnerVolumeSpecName "kube-api-access-9zbsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:55.013583 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.013521 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l" pod "946874be-d475-41a5-a53e-78a44039a0ad" (UID: "946874be-d475-41a5-a53e-78a44039a0ad"). InnerVolumeSpecName "kube-api-access-m6b4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:55.111948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.111923 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6b4l\" (UniqueName: \"kubernetes.io/projected/946874be-d475-41a5-a53e-78a44039a0ad-kube-api-access-m6b4l\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:55.111948 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.111946 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e90998-0ef9-42a6-aa06-e600ff01e97e-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:55.112364 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.111958 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zbsg\" (UniqueName: \"kubernetes.io/projected/a2e90998-0ef9-42a6-aa06-e600ff01e97e-kube-api-access-9zbsg\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:55.112364 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.111968 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946874be-d475-41a5-a53e-78a44039a0ad-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:55.581568 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.581534 4854 generic.go:356] "Generic (PLEG): container finished" podID="946874be-d475-41a5-a53e-78a44039a0ad" containerID="6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db" exitCode=0 Oct 31 03:36:55.581779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.581609 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" Oct 31 03:36:55.581779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.581617 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" event={"ID":"946874be-d475-41a5-a53e-78a44039a0ad","Type":"ContainerDied","Data":"6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db"} Oct 31 03:36:55.581779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.581673 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx" event={"ID":"946874be-d475-41a5-a53e-78a44039a0ad","Type":"ContainerDied","Data":"249723b31a19d51c16f5d51cf7fcb965f0c26e46235c43b9ff30c21bd314ece7"} Oct 31 03:36:55.581779 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.581696 4854 scope.go:122] "RemoveContainer" containerID="6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db" Oct 31 03:36:55.583080 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.583053 4854 generic.go:356] "Generic (PLEG): container finished" podID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerID="614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f" exitCode=0 Oct 31 03:36:55.583212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.583126 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" Oct 31 03:36:55.583212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.583142 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" event={"ID":"a2e90998-0ef9-42a6-aa06-e600ff01e97e","Type":"ContainerDied","Data":"614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f"} Oct 31 03:36:55.583352 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.583228 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm" event={"ID":"a2e90998-0ef9-42a6-aa06-e600ff01e97e","Type":"ContainerDied","Data":"d3d98318cfe5592b1a33921d2639d76a8f4a4fe4362ce96bc0962d96d804d5de"} Oct 31 03:36:55.592538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.592522 4854 scope.go:122] "RemoveContainer" containerID="6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db" Oct 31 03:36:55.592796 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:36:55.592776 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db\": container with ID starting with 6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db not found: ID does not exist" containerID="6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db" Oct 31 03:36:55.592899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.592802 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db"} err="failed to get container status \"6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db\": rpc error: code = NotFound desc = could not find container \"6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db\": container with ID starting with 6a451c6eef6043c2bf58ae2bb01d8979ffb808aff81301da196afb0cd89d48db not found: ID does not exist" Oct 31 03:36:55.592899 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.592819 4854 scope.go:122] "RemoveContainer" containerID="614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f" Oct 31 03:36:55.600725 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.600709 4854 scope.go:122] "RemoveContainer" containerID="614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f" Oct 31 03:36:55.600962 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:36:55.600944 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f\": container with ID starting with 614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f not found: ID does not exist" containerID="614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f" Oct 31 03:36:55.601009 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.600968 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f"} err="failed to get container status \"614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f\": rpc error: code = NotFound desc = could not find container \"614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f\": container with ID starting with 614158afe154c8a036c8eca09863e27af200e3253d084844d070eeeb46b98c5f not found: ID does not exist" Oct 31 03:36:55.603241 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.603222 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx"] Oct 31 03:36:55.609411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.609393 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-31e475-696b45845tpplx"] Oct 31 03:36:55.618558 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.618534 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm"] Oct 31 03:36:55.624532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:55.624511 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-31e475-5f5c9ps8fm"] Oct 31 03:36:56.455457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:56.455425 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946874be-d475-41a5-a53e-78a44039a0ad" path="/var/lib/kubelet/pods/946874be-d475-41a5-a53e-78a44039a0ad/volumes" Oct 31 03:36:56.457892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:36:56.455765 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" path="/var/lib/kubelet/pods/a2e90998-0ef9-42a6-aa06-e600ff01e97e/volumes" Oct 31 03:37:09.250864 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:37:09.250825 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:37:32.306274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:37:32.301342 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mcqz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:37:32.306274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:37:32.301423 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mcqz2" podUID="b46a699c-a9b7-4a4a-a8b3-43d37829903a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:37:39.252026 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:37:39.251986 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:37:43.613378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:37:43.613338 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:37:43.614023 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:37:43.613988 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:37:43.664274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:37:43.663815 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:37:43.664274 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:37:43.663883 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:08.655368 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655337 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm"] Oct 31 03:38:08.655767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655738 4854 cpu_manager.go:414] "containerMap: removing container" podUID="946874be-d475-41a5-a53e-78a44039a0ad" containerName="payload-processing" Oct 31 03:38:08.655767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655749 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946874be-d475-41a5-a53e-78a44039a0ad" containerName="payload-processing" Oct 31 03:38:08.655846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655774 4854 cpu_manager.go:414] "containerMap: removing container" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerName="payload-pre-processing" Oct 31 03:38:08.655846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655779 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerName="payload-pre-processing" Oct 31 03:38:08.655846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655837 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="946874be-d475-41a5-a53e-78a44039a0ad" containerName="payload-processing" Oct 31 03:38:08.655939 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.655874 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e90998-0ef9-42a6-aa06-e600ff01e97e" containerName="payload-pre-processing" Oct 31 03:38:08.659581 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.659561 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.662512 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.662479 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-17e11a7f-openshift-default-dockercfg-6lg27\"" Oct 31 03:38:08.674146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.674116 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm"] Oct 31 03:38:08.704910 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.704882 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.704954 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705094 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705073 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2jt\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705135 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705109 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705202 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705186 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705213 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705258 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705404 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705378 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.705439 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.705426 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806030 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.805992 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806057 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806137 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806171 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2jt\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806212 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806204 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806325 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806359 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806409 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806473 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806638 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806495 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806737 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806683 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.806985 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.806957 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.807170 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.807141 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.807317 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.807295 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.813017 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.812988 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.814056 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.814027 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.821411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.821385 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.821865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.821841 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2jt\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt\") pod \"e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:08.887289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.887262 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst"] Oct 31 03:38:08.892089 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.892070 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:08.909177 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.902801 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst"] Oct 31 03:38:08.972903 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:08.972877 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:09.008777 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.008742 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74fc6594-6ea0-4f48-866a-564809d7c789-metrics-certs\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.008915 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.008801 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fc6594-6ea0-4f48-866a-564809d7c789-openshift-service-ca\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.008915 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.008888 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74fc6594-6ea0-4f48-866a-564809d7c789-maas-api-tls\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.009037 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.008921 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzd8\" (UniqueName: \"kubernetes.io/projected/74fc6594-6ea0-4f48-866a-564809d7c789-kube-api-access-bgzd8\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.110052 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.110021 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74fc6594-6ea0-4f48-866a-564809d7c789-metrics-certs\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.110223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.110060 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fc6594-6ea0-4f48-866a-564809d7c789-openshift-service-ca\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.110223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.110084 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74fc6594-6ea0-4f48-866a-564809d7c789-maas-api-tls\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.110223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.110194 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzd8\" (UniqueName: \"kubernetes.io/projected/74fc6594-6ea0-4f48-866a-564809d7c789-kube-api-access-bgzd8\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.110782 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.110764 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fc6594-6ea0-4f48-866a-564809d7c789-openshift-service-ca\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.112635 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.112614 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74fc6594-6ea0-4f48-866a-564809d7c789-maas-api-tls\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.112732 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.112686 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74fc6594-6ea0-4f48-866a-564809d7c789-metrics-certs\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.121854 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.121835 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzd8\" (UniqueName: \"kubernetes.io/projected/74fc6594-6ea0-4f48-866a-564809d7c789-kube-api-access-bgzd8\") pod \"maas-api-57d9c7bd57-qfzst\" (UID: \"74fc6594-6ea0-4f48-866a-564809d7c789\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.208960 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.208902 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:09.256388 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.256363 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm"] Oct 31 03:38:09.259599 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:38:09.259571 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5471da67_bb9a_4813_8dc8_104dae2dabb0.slice/crio-dc2ec4cc5efb42f10f75c778a198efe6a21750ee70a001cfc5e83dd974d9f832 WatchSource:0}: Error finding container dc2ec4cc5efb42f10f75c778a198efe6a21750ee70a001cfc5e83dd974d9f832: Status 404 returned error can't find the container with id dc2ec4cc5efb42f10f75c778a198efe6a21750ee70a001cfc5e83dd974d9f832 Oct 31 03:38:09.264824 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:38:09.264772 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:38:09.265745 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.265705 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:38:09.265839 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.265807 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:38:09.265900 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.265855 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:38:09.491459 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.491433 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst"] Oct 31 03:38:09.493943 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:38:09.493914 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fc6594_6ea0_4f48_866a_564809d7c789.slice/crio-9ceee0d14308a7270bc6a5d60730a7f13dad956a65dead819e42712524d63a70 WatchSource:0}: Error finding container 9ceee0d14308a7270bc6a5d60730a7f13dad956a65dead819e42712524d63a70: Status 404 returned error can't find the container with id 9ceee0d14308a7270bc6a5d60730a7f13dad956a65dead819e42712524d63a70 Oct 31 03:38:09.495601 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:09.495577 4854 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:38:10.185072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.184978 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" event={"ID":"5471da67-bb9a-4813-8dc8-104dae2dabb0","Type":"ContainerStarted","Data":"aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454"} Oct 31 03:38:10.185072 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.185041 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" event={"ID":"5471da67-bb9a-4813-8dc8-104dae2dabb0","Type":"ContainerStarted","Data":"dc2ec4cc5efb42f10f75c778a198efe6a21750ee70a001cfc5e83dd974d9f832"} Oct 31 03:38:10.186722 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.186698 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" event={"ID":"74fc6594-6ea0-4f48-866a-564809d7c789","Type":"ContainerStarted","Data":"2f030e853ca9eb9efad9f03b8965b3c13f99c3b5290e60bd3bc201411c9f874d"} Oct 31 03:38:10.186819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.186738 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" event={"ID":"74fc6594-6ea0-4f48-866a-564809d7c789","Type":"ContainerStarted","Data":"9ceee0d14308a7270bc6a5d60730a7f13dad956a65dead819e42712524d63a70"} Oct 31 03:38:10.186819 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.186771 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:10.206147 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.206106 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podStartSLOduration=2.206093616 podStartE2EDuration="2.206093616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:10.204916016 +0000 UTC m=+2573.902083323" watchObservedRunningTime="2026-10-31 03:38:10.206093616 +0000 UTC m=+2573.903260920" Oct 31 03:38:10.223118 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.222446 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" podStartSLOduration=1.828033854 podStartE2EDuration="2.222430083s" totalImagesPullingTime="394.396229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:10.222330437 +0000 UTC m=+2573.919497753" watchObservedRunningTime="2026-10-31 03:38:10.222430083 +0000 UTC m=+2573.919597388" Oct 31 03:38:10.973222 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.973187 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:10.978532 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:10.978506 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:11.192560 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:11.192530 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:11.193679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:11.193655 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:16.219469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:16.219431 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-qfzst" Oct 31 03:38:16.274636 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:16.274604 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz"] Oct 31 03:38:16.274836 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:16.274815 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" podUID="7f6d1a73-179d-4794-bd39-169c04ced3f7" containerName="maas-api" containerID="cri-o://bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26" gracePeriod=30 Oct 31 03:38:16.588025 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:16.587983 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" podUID="7f6d1a73-179d-4794-bd39-169c04ced3f7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.33:8443/health\": dial tcp 10.132.0.33:8443: connect: connection refused" Oct 31 03:38:17.626808 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.626741 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:38:17.736735 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.736705 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs\") pod \"7f6d1a73-179d-4794-bd39-169c04ced3f7\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " Oct 31 03:38:17.736931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.736801 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd\" (UniqueName: \"kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd\") pod \"7f6d1a73-179d-4794-bd39-169c04ced3f7\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " Oct 31 03:38:17.736931 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.736906 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls\") pod \"7f6d1a73-179d-4794-bd39-169c04ced3f7\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " Oct 31 03:38:17.737079 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.736939 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca\") pod \"7f6d1a73-179d-4794-bd39-169c04ced3f7\" (UID: \"7f6d1a73-179d-4794-bd39-169c04ced3f7\") " Oct 31 03:38:17.737360 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.737331 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca" pod "7f6d1a73-179d-4794-bd39-169c04ced3f7" (UID: "7f6d1a73-179d-4794-bd39-169c04ced3f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:17.739136 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.739109 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs" pod "7f6d1a73-179d-4794-bd39-169c04ced3f7" (UID: "7f6d1a73-179d-4794-bd39-169c04ced3f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:17.739284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.739189 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls" pod "7f6d1a73-179d-4794-bd39-169c04ced3f7" (UID: "7f6d1a73-179d-4794-bd39-169c04ced3f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:17.739284 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.739187 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd" pod "7f6d1a73-179d-4794-bd39-169c04ced3f7" (UID: "7f6d1a73-179d-4794-bd39-169c04ced3f7"). InnerVolumeSpecName "kube-api-access-7p8zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:17.838330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.838297 4854 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-metrics-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:17.838330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.838325 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8zd\" (UniqueName: \"kubernetes.io/projected/7f6d1a73-179d-4794-bd39-169c04ced3f7-kube-api-access-7p8zd\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:17.838330 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.838338 4854 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f6d1a73-179d-4794-bd39-169c04ced3f7-maas-api-tls\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:17.838569 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:17.838347 4854 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f6d1a73-179d-4794-bd39-169c04ced3f7-openshift-service-ca\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:18.244490 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.244449 4854 generic.go:356] "Generic (PLEG): container finished" podID="7f6d1a73-179d-4794-bd39-169c04ced3f7" containerID="bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26" exitCode=0 Oct 31 03:38:18.244679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.244502 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" Oct 31 03:38:18.244679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.244540 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" event={"ID":"7f6d1a73-179d-4794-bd39-169c04ced3f7","Type":"ContainerDied","Data":"bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26"} Oct 31 03:38:18.244679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.244615 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz" event={"ID":"7f6d1a73-179d-4794-bd39-169c04ced3f7","Type":"ContainerDied","Data":"587c906a08749914db1e99dad240def392cbf38142a4c976aadbed696818dfbc"} Oct 31 03:38:18.244679 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.244647 4854 scope.go:122] "RemoveContainer" containerID="bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26" Oct 31 03:38:18.272912 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.272876 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz"] Oct 31 03:38:18.277300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.276882 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-hmtnz"] Oct 31 03:38:18.277300 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.277031 4854 scope.go:122] "RemoveContainer" containerID="bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26" Oct 31 03:38:18.278704 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:38:18.278677 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26\": container with ID starting with bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26 not found: ID does not exist" containerID="bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26" Oct 31 03:38:18.278841 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.278716 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26"} err="failed to get container status \"bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26\": rpc error: code = NotFound desc = could not find container \"bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26\": container with ID starting with bc52cd30c6aa4dabafe95939f17f923c73b98f55f37157489af4748ea5720c26 not found: ID does not exist" Oct 31 03:38:18.455497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:18.455462 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6d1a73-179d-4794-bd39-169c04ced3f7" path="/var/lib/kubelet/pods/7f6d1a73-179d-4794-bd39-169c04ced3f7/volumes" Oct 31 03:38:25.300835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:25.300797 4854 generic.go:356] "Generic (PLEG): container finished" podID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerID="aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454" exitCode=0 Oct 31 03:38:25.301410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:25.300877 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" event={"ID":"5471da67-bb9a-4813-8dc8-104dae2dabb0","Type":"ContainerDied","Data":"aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454"} Oct 31 03:38:25.301410 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:25.301382 4854 scope.go:122] "RemoveContainer" containerID="aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454" Oct 31 03:38:25.302589 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:25.302561 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:38:25.302680 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:25.302628 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:38:25.302680 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:25.302659 4854 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:38:26.200457 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:26.200358 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:26.319053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:26.319019 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" event={"ID":"5471da67-bb9a-4813-8dc8-104dae2dabb0","Type":"ContainerStarted","Data":"0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d"} Oct 31 03:38:26.973756 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:26.973723 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:26.974913 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:26.974890 4854 patch_prober.go:29] interesting pod/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:38:26.975022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:26.974941 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 03:38:27.973749 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:27.973719 4854 patch_prober.go:29] interesting pod/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:38:27.974169 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:27.973772 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 03:38:28.973824 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:28.973780 4854 patch_prober.go:29] interesting pod/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:38:28.974223 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:28.973851 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 03:38:29.973698 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:29.973662 4854 patch_prober.go:29] interesting pod/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:38:29.973905 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:29.973715 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 03:38:30.973266 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:30.973221 4854 patch_prober.go:29] interesting pod/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:38:30.973460 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:30.973307 4854 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 03:38:31.978117 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:31.978086 4854 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:31.978597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:31.978362 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:31.979175 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:31.979159 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:39.250944 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:38:39.250904 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:38:44.159180 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:44.159138 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm"] Oct 31 03:38:44.159662 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:44.159424 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" containerID="cri-o://0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d" gracePeriod=30 Oct 31 03:38:46.980871 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:46.980828 4854 patch_prober.go:29] interesting pod/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:38:46.981280 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:46.980889 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:38:49.431594 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.431529 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:49.436731 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.436704 4854 generic.go:356] "Generic (PLEG): container finished" podID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerID="0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d" exitCode=0 Oct 31 03:38:49.436848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.436750 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" Oct 31 03:38:49.436848 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.436787 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" event={"ID":"5471da67-bb9a-4813-8dc8-104dae2dabb0","Type":"ContainerDied","Data":"0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d"} Oct 31 03:38:49.436935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.436849 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm" event={"ID":"5471da67-bb9a-4813-8dc8-104dae2dabb0","Type":"ContainerDied","Data":"dc2ec4cc5efb42f10f75c778a198efe6a21750ee70a001cfc5e83dd974d9f832"} Oct 31 03:38:49.436935 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.436878 4854 scope.go:122] "RemoveContainer" containerID="0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d" Oct 31 03:38:49.445497 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.445481 4854 scope.go:122] "RemoveContainer" containerID="aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454" Oct 31 03:38:49.453869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.453771 4854 scope.go:122] "RemoveContainer" containerID="0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d" Oct 31 03:38:49.454083 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:38:49.454049 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d\": container with ID starting with 0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d not found: ID does not exist" containerID="0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d" Oct 31 03:38:49.454196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.454085 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d"} err="failed to get container status \"0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d\": rpc error: code = NotFound desc = could not find container \"0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d\": container with ID starting with 0e13e58a9defcae6f70ec01bd1e01f03dfbeaf937fea7099bad110a0a2bec02d not found: ID does not exist" Oct 31 03:38:49.454196 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.454105 4854 scope.go:122] "RemoveContainer" containerID="aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454" Oct 31 03:38:49.454463 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:38:49.454435 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454\": container with ID starting with aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454 not found: ID does not exist" containerID="aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454" Oct 31 03:38:49.454565 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.454497 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454"} err="failed to get container status \"aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454\": rpc error: code = NotFound desc = could not find container \"aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454\": container with ID starting with aa5d38529bf7c8de7364619a1077b3c8074696f3109c8e497670cea2c31b5454 not found: ID does not exist" Oct 31 03:38:49.581296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581240 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581336 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581394 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581461 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581423 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581467 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581523 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:38:49.581574 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581563 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581613 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581695 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581732 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:38:49.581846 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581766 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket\") pod \"5471da67-bb9a-4813-8dc8-104dae2dabb0\" (UID: \"5471da67-bb9a-4813-8dc8-104dae2dabb0\") " Oct 31 03:38:49.581907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581842 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:38:49.581907 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.581851 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:49.582083 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.582064 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:38:49.582167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.582150 4854 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.582220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.582174 4854 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-credential-socket\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.582220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.582188 4854 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5471da67-bb9a-4813-8dc8-104dae2dabb0-istiod-ca-cert\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.582220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.582201 4854 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-workload-certs\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.582220 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.582214 4854 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-data\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.584413 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.584386 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "kube-api-access-wx2jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:49.584538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.584422 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:49.584538 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.584525 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:38:49.584612 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.584600 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy" pod "5471da67-bb9a-4813-8dc8-104dae2dabb0" (UID: "5471da67-bb9a-4813-8dc8-104dae2dabb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:38:49.683530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.683474 4854 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-envoy\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.683530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.683497 4854 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-podinfo\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.683530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.683507 4854 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-istio-token\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.683530 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.683516 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2jt\" (UniqueName: \"kubernetes.io/projected/5471da67-bb9a-4813-8dc8-104dae2dabb0-kube-api-access-wx2jt\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:49.757981 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.757950 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm"] Oct 31 03:38:49.762670 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:49.762646 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-17e11a7f-openshift-default-85fd668947-qv7pm"] Oct 31 03:38:50.455894 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.455861 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" path="/var/lib/kubelet/pods/5471da67-bb9a-4813-8dc8-104dae2dabb0/volumes" Oct 31 03:38:50.496032 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.495995 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8"] Oct 31 03:38:50.496435 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496422 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" Oct 31 03:38:50.496502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496435 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" Oct 31 03:38:50.496502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496459 4854 cpu_manager.go:414] "containerMap: removing container" podUID="7f6d1a73-179d-4794-bd39-169c04ced3f7" containerName="maas-api" Oct 31 03:38:50.496502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496464 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6d1a73-179d-4794-bd39-169c04ced3f7" containerName="maas-api" Oct 31 03:38:50.496502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496473 4854 cpu_manager.go:414] "containerMap: removing container" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" Oct 31 03:38:50.496502 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496478 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" Oct 31 03:38:50.496659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496533 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" Oct 31 03:38:50.496659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496545 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6d1a73-179d-4794-bd39-169c04ced3f7" containerName="maas-api" Oct 31 03:38:50.496659 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.496552 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="5471da67-bb9a-4813-8dc8-104dae2dabb0" containerName="istio-proxy" Oct 31 03:38:50.501613 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.501589 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.508847 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.508823 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-8d20d2\"" Oct 31 03:38:50.512710 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.512683 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8"] Oct 31 03:38:50.564537 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.564508 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc"] Oct 31 03:38:50.568378 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.568358 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.578428 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.578389 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-8d20d2-dockercfg-4wggg\"" Oct 31 03:38:50.579873 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.579843 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc"] Oct 31 03:38:50.592988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.592963 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.593132 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.593028 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgc4\" (UniqueName: \"kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4\") pod \"payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.696016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.693715 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdf4v\" (UniqueName: \"kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v\") pod \"payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.696016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.693828 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.696016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.693878 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.696016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.693948 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgc4\" (UniqueName: \"kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4\") pod \"payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.696016 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.695237 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.705445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.705421 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgc4\" (UniqueName: \"kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4\") pod \"payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.794643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.794613 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdf4v\" (UniqueName: \"kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v\") pod \"payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.794789 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.794670 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.795384 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.795366 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.805588 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.805563 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdf4v\" (UniqueName: \"kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v\") pod \"payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:50.818350 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.818329 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:50.885479 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:50.885450 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:51.092504 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.092474 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8"] Oct 31 03:38:51.094221 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:38:51.094191 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6017cbfa_e522_4fe9_b7f0_d2a7dd1a0092.slice/crio-21af8c90665d6a13961af75eeeacb22a3dcb4b89db0dfc08f3ed3f4ac4e84bde WatchSource:0}: Error finding container 21af8c90665d6a13961af75eeeacb22a3dcb4b89db0dfc08f3ed3f4ac4e84bde: Status 404 returned error can't find the container with id 21af8c90665d6a13961af75eeeacb22a3dcb4b89db0dfc08f3ed3f4ac4e84bde Oct 31 03:38:51.192392 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.192367 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc"] Oct 31 03:38:51.193145 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:38:51.193117 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497ae085_8bc2_4ae8_b61c_bb349ece60b3.slice/crio-ee5451ba5ba0149d0ccdaa0a249083f233bcd12b93d0eae5c91c2ec011a1aafa WatchSource:0}: Error finding container ee5451ba5ba0149d0ccdaa0a249083f233bcd12b93d0eae5c91c2ec011a1aafa: Status 404 returned error can't find the container with id ee5451ba5ba0149d0ccdaa0a249083f233bcd12b93d0eae5c91c2ec011a1aafa Oct 31 03:38:51.451643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.451549 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" event={"ID":"497ae085-8bc2-4ae8-b61c-bb349ece60b3","Type":"ContainerStarted","Data":"9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2"} Oct 31 03:38:51.451643 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.451599 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" event={"ID":"497ae085-8bc2-4ae8-b61c-bb349ece60b3","Type":"ContainerStarted","Data":"ee5451ba5ba0149d0ccdaa0a249083f233bcd12b93d0eae5c91c2ec011a1aafa"} Oct 31 03:38:51.451879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.451682 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:38:51.453202 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.453162 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" event={"ID":"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092","Type":"ContainerStarted","Data":"8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d"} Oct 31 03:38:51.453373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.453220 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" event={"ID":"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092","Type":"ContainerStarted","Data":"21af8c90665d6a13961af75eeeacb22a3dcb4b89db0dfc08f3ed3f4ac4e84bde"} Oct 31 03:38:51.453373 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.453279 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:38:51.482059 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.472430 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" podStartSLOduration=1.472413709 podStartE2EDuration="1.472413709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:51.467363438 +0000 UTC m=+2615.164530776" watchObservedRunningTime="2026-10-31 03:38:51.472413709 +0000 UTC m=+2615.169581014" Oct 31 03:38:51.487134 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:38:51.487075 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" podStartSLOduration=1.487060325 podStartE2EDuration="1.487060325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:51.484985 +0000 UTC m=+2615.182152304" watchObservedRunningTime="2026-10-31 03:38:51.487060325 +0000 UTC m=+2615.184227628" Oct 31 03:39:02.460627 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:39:02.460601 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:39:02.461011 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:39:02.460667 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:39:09.250411 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:39:09.250374 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:39:39.249636 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:39:39.249554 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:40:09.263056 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:40:09.263013 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:40:19.838954 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:19.838927 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:40:19.847730 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:19.847705 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:40:35.028523 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:35.028484 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc"] Oct 31 03:40:35.038289 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:35.037202 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8"] Oct 31 03:40:39.249941 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:40:39.249900 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:40:40.029828 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.029762 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" podUID="497ae085-8bc2-4ae8-b61c-bb349ece60b3" containerName="payload-processing" containerID="cri-o://9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2" gracePeriod=25 Oct 31 03:40:40.038266 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.038215 4854 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" podUID="6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" containerName="payload-pre-processing" containerID="cri-o://8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d" gracePeriod=25 Oct 31 03:40:40.287351 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.287298 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:40:40.313772 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.313744 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:40:40.342076 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.342047 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume\") pod \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " Oct 31 03:40:40.342201 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.342120 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v\" (UniqueName: \"kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v\") pod \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\" (UID: \"497ae085-8bc2-4ae8-b61c-bb349ece60b3\") " Oct 31 03:40:40.342465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.342445 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume" pod "497ae085-8bc2-4ae8-b61c-bb349ece60b3" (UID: "497ae085-8bc2-4ae8-b61c-bb349ece60b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:40.344296 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.344271 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v" pod "497ae085-8bc2-4ae8-b61c-bb349ece60b3" (UID: "497ae085-8bc2-4ae8-b61c-bb349ece60b3"). InnerVolumeSpecName "kube-api-access-hdf4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:40.442811 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.442789 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4\" (UniqueName: \"kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4\") pod \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " Oct 31 03:40:40.442933 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.442920 4854 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume\") pod \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\" (UID: \"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092\") " Oct 31 03:40:40.443101 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.443088 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497ae085-8bc2-4ae8-b61c-bb349ece60b3-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:40.443142 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.443104 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdf4v\" (UniqueName: \"kubernetes.io/projected/497ae085-8bc2-4ae8-b61c-bb349ece60b3-kube-api-access-hdf4v\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:40.443314 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.443294 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume" pod "6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" (UID: "6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:40.444925 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.444909 4854 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4" pod "6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" (UID: "6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092"). InnerVolumeSpecName "kube-api-access-cfgc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:40.544511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.544449 4854 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-plugins-config-volume\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:40.544511 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:40.544478 4854 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgc4\" (UniqueName: \"kubernetes.io/projected/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092-kube-api-access-cfgc4\") on node \"ip-10-0-130-12.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:41.016417 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.016380 4854 generic.go:356] "Generic (PLEG): container finished" podID="497ae085-8bc2-4ae8-b61c-bb349ece60b3" containerID="9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2" exitCode=0 Oct 31 03:40:41.016597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.016434 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" Oct 31 03:40:41.016597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.016472 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" event={"ID":"497ae085-8bc2-4ae8-b61c-bb349ece60b3","Type":"ContainerDied","Data":"9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2"} Oct 31 03:40:41.016597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.016531 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc" event={"ID":"497ae085-8bc2-4ae8-b61c-bb349ece60b3","Type":"ContainerDied","Data":"ee5451ba5ba0149d0ccdaa0a249083f233bcd12b93d0eae5c91c2ec011a1aafa"} Oct 31 03:40:41.016597 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.016560 4854 scope.go:122] "RemoveContainer" containerID="9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2" Oct 31 03:40:41.017963 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.017937 4854 generic.go:356] "Generic (PLEG): container finished" podID="6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" containerID="8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d" exitCode=0 Oct 31 03:40:41.018096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.017978 4854 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" Oct 31 03:40:41.018096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.018024 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" event={"ID":"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092","Type":"ContainerDied","Data":"8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d"} Oct 31 03:40:41.018096 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.018075 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8" event={"ID":"6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092","Type":"ContainerDied","Data":"21af8c90665d6a13961af75eeeacb22a3dcb4b89db0dfc08f3ed3f4ac4e84bde"} Oct 31 03:40:41.026160 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.026142 4854 scope.go:122] "RemoveContainer" containerID="9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2" Oct 31 03:40:41.026485 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:40:41.026463 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2\": container with ID starting with 9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2 not found: ID does not exist" containerID="9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2" Oct 31 03:40:41.026564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.026492 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2"} err="failed to get container status \"9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2\": rpc error: code = NotFound desc = could not find container \"9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2\": container with ID starting with 9b6a8c34622e18220003e48cf84c5836003c57658cf52f39e3e783d90bf803e2 not found: ID does not exist" Oct 31 03:40:41.026564 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.026509 4854 scope.go:122] "RemoveContainer" containerID="8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d" Oct 31 03:40:41.034685 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.034529 4854 scope.go:122] "RemoveContainer" containerID="8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d" Oct 31 03:40:41.034841 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:40:41.034820 4854 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d\": container with ID starting with 8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d not found: ID does not exist" containerID="8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d" Oct 31 03:40:41.034891 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.034849 4854 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d"} err="failed to get container status \"8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d\": rpc error: code = NotFound desc = could not find container \"8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d\": container with ID starting with 8583a9c5477826fa846370b95c794252fdc0e82d4958b9c405de6e3abc17a88d not found: ID does not exist" Oct 31 03:40:41.035381 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.035364 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc"] Oct 31 03:40:41.042174 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.042155 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8d20d2-776d465db7-r7nxc"] Oct 31 03:40:41.050865 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.050846 4854 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8"] Oct 31 03:40:41.054988 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:41.054960 4854 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d20d2-67cb59cf59-vnzx8"] Oct 31 03:40:42.455407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:42.455374 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497ae085-8bc2-4ae8-b61c-bb349ece60b3" path="/var/lib/kubelet/pods/497ae085-8bc2-4ae8-b61c-bb349ece60b3/volumes" Oct 31 03:40:42.455914 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:42.455897 4854 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" path="/var/lib/kubelet/pods/6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092/volumes" Oct 31 03:40:58.600324 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:58.600206 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7v4sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:58.601167 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:58.601133 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7v4sj" podUID="28f78da7-dfdb-43d8-8062-86bb0e8daba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:58.662277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:58.661609 4854 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lp4mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:58.662277 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:40:58.661672 4854 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lp4mf" podUID="5facc9ef-64c0-46b2-8d88-75e73722a799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:09.249929 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:41:09.249889 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:41:39.378131 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:41:39.378082 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:41:49.268726 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:49.268692 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79bd86dfd9-966ss_7a92420e-917f-40b9-8b4c-2095dc028257/manager/0.log" Oct 31 03:41:50.183453 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:50.183422 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64564ccd46-thk28_cff88014-ac5d-4d2d-97a6-6383a4a619af/manager/0.log" Oct 31 03:41:50.374691 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:50.374667 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7bvdz_7d599f3d-829f-4268-beb8-739eb59882ec/server/0.log" Oct 31 03:41:50.558153 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:50.558128 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-7tcqw_f59247df-efcb-4217-8b5a-66282ba7d89f/manager/0.log" Oct 31 03:41:51.801034 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:51.800996 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57d9c7bd57-qfzst_74fc6594-6ea0-4f48-866a-564809d7c789/maas-api/0.log" Oct 31 03:41:55.343861 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:55.343833 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr84g_997f197c-63bc-4bfc-b84a-6e620cea33e3/kuadrant-console-plugin/0.log" Oct 31 03:41:55.964589 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:55.964535 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rlp29_763496b7-9cc7-475a-96c9-86a30d8b68c5/manager/0.log" Oct 31 03:41:56.983452 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:56.983421 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dbq78_a5461b19-2316-454f-a955-73429d2e50a2/discovery/0.log" Oct 31 03:41:57.779046 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:57.779007 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7v4sj_28f78da7-dfdb-43d8-8062-86bb0e8daba1/istio-proxy/0.log" Oct 31 03:41:58.193406 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:58.193326 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-kc577_3621ef10-eb7d-4e57-9c29-5f3598cae2d9/istio-proxy/0.log" Oct 31 03:41:58.403441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:58.403416 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lp4mf_5facc9ef-64c0-46b2-8d88-75e73722a799/istio-proxy/0.log" Oct 31 03:41:58.596956 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:58.596929 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mcqz2_b46a699c-a9b7-4a4a-a8b3-43d37829903a/istio-proxy/0.log" Oct 31 03:41:59.954493 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:41:59.954464 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:42:00.146969 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:00.146943 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66695d6bd7-5pk65_bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/router/0.log" Oct 31 03:42:01.145021 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:01.144986 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5_94f7442a-d473-49cc-aa44-0cb836085273/main/0.log" Oct 31 03:42:01.504886 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:01.504850 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wmfx5_c2b332bc-d96f-4af7-871e-be4a9a439e9a/main/0.log" Oct 31 03:42:02.056704 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:02.056676 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jlgpj_9ca899b4-82ad-4c0f-9654-f9912c269a4e/main/0.log" Oct 31 03:42:02.245036 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:02.245008 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt_c679a142-f04f-4b6f-b25e-ed237eb68f78/main/0.log" Oct 31 03:42:09.250558 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:42:09.250518 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:42:13.139625 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:13.139599 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79bd86dfd9-966ss_7a92420e-917f-40b9-8b4c-2095dc028257/manager/0.log" Oct 31 03:42:14.050874 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:14.050846 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64564ccd46-thk28_cff88014-ac5d-4d2d-97a6-6383a4a619af/manager/0.log" Oct 31 03:42:14.277454 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:14.277425 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7bvdz_7d599f3d-829f-4268-beb8-739eb59882ec/server/0.log" Oct 31 03:42:14.469947 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:14.469870 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-7tcqw_f59247df-efcb-4217-8b5a-66282ba7d89f/manager/0.log" Oct 31 03:42:15.471831 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:15.471797 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57d9c7bd57-qfzst_74fc6594-6ea0-4f48-866a-564809d7c789/maas-api/0.log" Oct 31 03:42:18.821654 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:18.821591 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr84g_997f197c-63bc-4bfc-b84a-6e620cea33e3/kuadrant-console-plugin/0.log" Oct 31 03:42:19.395146 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:19.395115 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rlp29_763496b7-9cc7-475a-96c9-86a30d8b68c5/manager/0.log" Oct 31 03:42:20.237469 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:20.237433 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dbq78_a5461b19-2316-454f-a955-73429d2e50a2/discovery/0.log" Oct 31 03:42:21.083766 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:21.083718 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7v4sj_28f78da7-dfdb-43d8-8062-86bb0e8daba1/istio-proxy/0.log" Oct 31 03:42:21.522983 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:21.522951 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-kc577_3621ef10-eb7d-4e57-9c29-5f3598cae2d9/istio-proxy/0.log" Oct 31 03:42:21.714910 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:21.714874 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lp4mf_5facc9ef-64c0-46b2-8d88-75e73722a799/istio-proxy/0.log" Oct 31 03:42:21.945951 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:21.945878 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mcqz2_b46a699c-a9b7-4a4a-a8b3-43d37829903a/istio-proxy/0.log" Oct 31 03:42:23.454445 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:23.454411 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:42:23.664499 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:23.664466 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66695d6bd7-5pk65_bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/router/0.log" Oct 31 03:42:24.246833 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:24.246803 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-txrg5_94f7442a-d473-49cc-aa44-0cb836085273/main/0.log" Oct 31 03:42:24.642798 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:24.642725 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wmfx5_c2b332bc-d96f-4af7-871e-be4a9a439e9a/main/0.log" Oct 31 03:42:25.262744 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:25.262717 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jlgpj_9ca899b4-82ad-4c0f-9654-f9912c269a4e/main/0.log" Oct 31 03:42:25.458792 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:42:25.458765 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rrlrt_c679a142-f04f-4b6f-b25e-ed237eb68f78/main/0.log" Oct 31 03:42:39.250522 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:42:39.250432 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:43:04.820889 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:04.820856 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64564ccd46-thk28_cff88014-ac5d-4d2d-97a6-6383a4a619af/manager/0.log" Oct 31 03:43:05.094002 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:05.093920 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64564ccd46-thk28_cff88014-ac5d-4d2d-97a6-6383a4a619af/manager/0.log" Oct 31 03:43:05.817411 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:05.817382 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:43:06.004019 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:06.003992 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:43:09.249609 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:43:09.249575 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:43:39.249751 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:43:39.249714 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:43:54.277586 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:54.277553 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64564ccd46-thk28_cff88014-ac5d-4d2d-97a6-6383a4a619af/manager/0.log" Oct 31 03:43:54.548708 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:54.548631 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64564ccd46-thk28_cff88014-ac5d-4d2d-97a6-6383a4a619af/manager/0.log" Oct 31 03:43:55.283239 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:55.283211 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:43:55.466875 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:55.466842 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:43:59.061066 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:59.060979 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4pp9z_566be63b-5b2b-4f4f-a40f-1e83e7837f74/global-pull-secret-syncer/0.log" Oct 31 03:43:59.298268 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:59.298216 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8mfvv_e8f27649-b353-4181-bc11-3aa1ec714be3/connection-checker/0.log" Oct 31 03:43:59.326892 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:59.326832 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9n9mr_94b41492-1a35-4ede-adcd-65e4c41c6cc0/connection-checker/0.log" Oct 31 03:43:59.355573 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:59.355552 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j5zbk_c4987385-e27b-4220-92d4-2f61bdb1ebc8/connection-checker/0.log" Oct 31 03:43:59.529367 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:59.529337 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sjjkf_64b281bc-7922-46b5-8d33-2f1bfdebb8ec/konnectivity-agent/0.log" Oct 31 03:43:59.594485 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:43:59.594414 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-12.us-east-2.compute.internal_915a6fb768d2fd4c56c8c1c0d6da80b2/haproxy/0.log" Oct 31 03:44:06.032107 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:06.032079 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jr84g_997f197c-63bc-4bfc-b84a-6e620cea33e3/kuadrant-console-plugin/0.log" Oct 31 03:44:06.242203 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:06.242174 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rlp29_763496b7-9cc7-475a-96c9-86a30d8b68c5/manager/0.log" Oct 31 03:44:07.934821 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:07.934785 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-fr9t2_6c03b70f-c1cc-4eb6-8cb6-10ab3a3cd1d5/cluster-monitoring-operator/0.log" Oct 31 03:44:08.373100 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:08.373069 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlk6q_e1e4dc82-af0b-49a3-b398-b9a7caafd206/node-exporter/0.log" Oct 31 03:44:08.412137 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:08.412107 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlk6q_e1e4dc82-af0b-49a3-b398-b9a7caafd206/kube-rbac-proxy/0.log" Oct 31 03:44:08.439465 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:08.439436 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlk6q_e1e4dc82-af0b-49a3-b398-b9a7caafd206/init-textfile/0.log" Oct 31 03:44:09.156837 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.156803 4854 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl"] Oct 31 03:44:09.157275 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.157261 4854 cpu_manager.go:414] "containerMap: removing container" podUID="6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" containerName="payload-pre-processing" Oct 31 03:44:09.157333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.157277 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" containerName="payload-pre-processing" Oct 31 03:44:09.157333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.157305 4854 cpu_manager.go:414] "containerMap: removing container" podUID="497ae085-8bc2-4ae8-b61c-bb349ece60b3" containerName="payload-processing" Oct 31 03:44:09.157333 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.157310 4854 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497ae085-8bc2-4ae8-b61c-bb349ece60b3" containerName="payload-processing" Oct 31 03:44:09.157441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.157393 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="497ae085-8bc2-4ae8-b61c-bb349ece60b3" containerName="payload-processing" Oct 31 03:44:09.157441 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.157408 4854 memory_manager.go:365] "RemoveStaleState removing state" podUID="6017cbfa-e522-4fe9-b7f0-d2a7dd1a0092" containerName="payload-pre-processing" Oct 31 03:44:09.160739 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.160719 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.164238 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.164220 4854 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68h4s\"/\"default-dockercfg-sdxsf\"" Oct 31 03:44:09.167053 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.167032 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68h4s\"/\"openshift-service-ca.crt\"" Oct 31 03:44:09.167214 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.167132 4854 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68h4s\"/\"kube-root-ca.crt\"" Oct 31 03:44:09.172266 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.172233 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl"] Oct 31 03:44:09.229451 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.229425 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-lib-modules\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.229602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.229476 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-proc\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.229602 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.229497 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-podres\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.229700 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.229675 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w25f\" (UniqueName: \"kubernetes.io/projected/476608a0-53b9-49db-939e-4307a3d6b897-kube-api-access-5w25f\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.229752 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.229737 4854 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-sys\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.250190 ip-10-0-130-12 kubenswrapper[4854]: E1031 03:44:09.250159 4854 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rsvkk" containerName="extractor" Oct 31 03:44:09.330888 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.330866 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-lib-modules\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.330995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.330906 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-proc\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.330995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.330930 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-podres\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.330995 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.330987 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-lib-modules\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.331103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.330990 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w25f\" (UniqueName: \"kubernetes.io/projected/476608a0-53b9-49db-939e-4307a3d6b897-kube-api-access-5w25f\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.331103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.330999 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-proc\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.331103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.331062 4854 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-sys\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.331103 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.331094 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-sys\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.331228 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.331065 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/476608a0-53b9-49db-939e-4307a3d6b897-podres\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.342476 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.342458 4854 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w25f\" (UniqueName: \"kubernetes.io/projected/476608a0-53b9-49db-939e-4307a3d6b897-kube-api-access-5w25f\") pod \"perf-node-gather-daemonset-bkrhl\" (UID: \"476608a0-53b9-49db-939e-4307a3d6b897\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.474599 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.474545 4854 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:09.748429 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:09.748406 4854 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl"] Oct 31 03:44:09.750817 ip-10-0-130-12 kubenswrapper[4854]: W1031 03:44:09.750793 4854 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod476608a0_53b9_49db_939e_4307a3d6b897.slice/crio-114bc2236e4c0a8ea7b24670e402c2b46206657b392ed6c440e396467015f3c9 WatchSource:0}: Error finding container 114bc2236e4c0a8ea7b24670e402c2b46206657b392ed6c440e396467015f3c9: Status 404 returned error can't find the container with id 114bc2236e4c0a8ea7b24670e402c2b46206657b392ed6c440e396467015f3c9 Oct 31 03:44:10.116593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:10.116501 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" event={"ID":"476608a0-53b9-49db-939e-4307a3d6b897","Type":"ContainerStarted","Data":"2cfbad0a207917e2f1c5cb63290e63ab7edc18ed1db11aae8296a3c10ddb8723"} Oct 31 03:44:10.116593 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:10.116569 4854 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" event={"ID":"476608a0-53b9-49db-939e-4307a3d6b897","Type":"ContainerStarted","Data":"114bc2236e4c0a8ea7b24670e402c2b46206657b392ed6c440e396467015f3c9"} Oct 31 03:44:10.116812 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:10.116711 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:10.135653 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:10.135602 4854 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" podStartSLOduration=1.135588715 podStartE2EDuration="1.135588715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:10.135382883 +0000 UTC m=+2933.832550191" watchObservedRunningTime="2026-10-31 03:44:10.135588715 +0000 UTC m=+2933.832756019" Oct 31 03:44:10.417171 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:10.417100 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-5k4sc_48eda75e-e883-4929-8278-fd49da0f5b27/networking-console-plugin/0.log" Oct 31 03:44:11.164705 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:11.164676 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/3.log" Oct 31 03:44:11.172099 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:11.172068 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8j7j_ec216fff-d06e-4a0a-b35e-73b3a2d513a8/console-operator/4.log" Oct 31 03:44:12.478188 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:12.478154 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xgpqk_05ce07aa-7ab8-45da-85c1-f036dc52c6b8/volume-data-source-validator/0.log" Oct 31 03:44:13.721855 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:13.721819 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwpgn_b3be8a54-63a4-408c-ae91-e21314a80325/dns/0.log" Oct 31 03:44:13.746301 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:13.746280 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwpgn_b3be8a54-63a4-408c-ae91-e21314a80325/kube-rbac-proxy/0.log" Oct 31 03:44:13.834020 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:13.833995 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ltlq_5e948ca8-1809-4eeb-b014-f563cb6fdb91/dns-node-resolver/0.log" Oct 31 03:44:14.531341 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:14.531307 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8nv7d_b408b194-9df2-4518-b247-0d9a83d80f3b/node-ca/0.log" Oct 31 03:44:16.135191 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:16.135159 4854 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-bkrhl" Oct 31 03:44:19.471851 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:19.471759 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dbq78_a5461b19-2316-454f-a955-73429d2e50a2/discovery/0.log" Oct 31 03:44:20.040092 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:20.040059 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7v4sj_28f78da7-dfdb-43d8-8062-86bb0e8daba1/istio-proxy/0.log" Oct 31 03:44:20.108226 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:20.108195 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-kc577_3621ef10-eb7d-4e57-9c29-5f3598cae2d9/istio-proxy/0.log" Oct 31 03:44:20.141767 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:20.141737 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lp4mf_5facc9ef-64c0-46b2-8d88-75e73722a799/istio-proxy/0.log" Oct 31 03:44:20.178442 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:20.178413 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mcqz2_b46a699c-a9b7-4a4a-a8b3-43d37829903a/istio-proxy/0.log" Oct 31 03:44:20.408879 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:20.408799 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:44:20.447525 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:20.447495 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66695d6bd7-5pk65_bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/router/0.log" Oct 31 03:44:22.245022 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:22.244986 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ss5vz_fb123961-0503-4503-b9d4-6790d658e12e/serve-healthcheck-canary/0.log" Oct 31 03:44:22.758804 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:22.758758 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-xth74_e7194787-0fa9-43a4-a749-c6aac9433cfc/insights-operator/0.log" Oct 31 03:44:22.761975 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:22.761946 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-xth74_e7194787-0fa9-43a4-a749-c6aac9433cfc/insights-operator/1.log" Oct 31 03:44:22.965869 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:22.965845 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsvkk_84351d2c-2829-4e33-afa0-0f3f813dd4fa/kube-rbac-proxy/0.log" Oct 31 03:44:22.993562 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:22.993537 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsvkk_84351d2c-2829-4e33-afa0-0f3f813dd4fa/exporter/0.log" Oct 31 03:44:23.022835 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.022772 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rsvkk_84351d2c-2829-4e33-afa0-0f3f813dd4fa/extractor/0.log" Oct 31 03:44:23.074563 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.074536 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-wwvvn-76z5v_cbff375c-6137-4767-b8e8-d8aa057f3e77/insights-gathering/0.log" Oct 31 03:44:23.461606 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.461524 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-dbq78_a5461b19-2316-454f-a955-73429d2e50a2/discovery/0.log" Oct 31 03:44:23.540866 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.540837 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-7v4sj_28f78da7-dfdb-43d8-8062-86bb0e8daba1/istio-proxy/0.log" Oct 31 03:44:23.584407 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.584382 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-kc577_3621ef10-eb7d-4e57-9c29-5f3598cae2d9/istio-proxy/0.log" Oct 31 03:44:23.603295 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.603270 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lp4mf_5facc9ef-64c0-46b2-8d88-75e73722a799/istio-proxy/0.log" Oct 31 03:44:23.621648 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.621624 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mcqz2_b46a699c-a9b7-4a4a-a8b3-43d37829903a/istio-proxy/0.log" Oct 31 03:44:23.766878 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.766851 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7hkl8_04a631e7-c686-4a5a-98ef-e963e34cd85a/payload-processing/0.log" Oct 31 03:44:23.790796 ip-10-0-130-12 kubenswrapper[4854]: I1031 03:44:23.790772 4854 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66695d6bd7-5pk65_bfdd5bc7-bedb-4abc-96ec-9e7d9f803702/router/0.log"